2023-12-07T23:25:31.073 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-07T23:25:31.073 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564 --verbose --name yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/orig.config.yaml 2023-12-07T23:25:31.115 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-07T23:25:31.204 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564 branch: wip-yuri3-testing-2023-12-07-0727 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7482564' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri3-testing-2023-12-07-0727 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-12-07-0727 sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9380 sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-12-07-0727 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk7r/sl5FVd4xb7HmAkal/vir+w88NJs5R3zxla6Lu2/yqjGtBg3JQ18Zye8EGgSE2uX+NVcIwyM+LRpgdP3FM= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA+p96ecYG4L+/ocaIhrFJzRHy5EQHa+ckbt4N23xqMZTSH7vVmpyFuMTVTnlnXutsIT1XFEzStsPoiZsgnm2k= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-07_21:35:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-07T23:25:31.204 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa; will attempt to use it 2023-12-07T23:25:31.205 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks 2023-12-07T23:25:31.205 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-07T23:25:31.207 INFO:teuthology.task.internal:Checking packages... 2023-12-07T23:25:31.231 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe' 2023-12-07T23:25:31.231 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-07T23:25:31.231 INFO:teuthology.packaging:ref: None 2023-12-07T23:25:31.231 INFO:teuthology.packaging:tag: None 2023-12-07T23:25:31.231 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-12-07-0727 2023-12-07T23:25:31.231 INFO:teuthology.packaging:sha1: c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-07T23:25:31.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri3-testing-2023-12-07-0727 2023-12-07T23:25:31.370 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-83.gc56ad3ea 2023-12-07T23:25:31.370 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-07T23:25:31.379 INFO:teuthology.task.internal:no buildpackages task found 2023-12-07T23:25:31.379 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-07T23:25:31.479 INFO:teuthology.task.internal:Saving configuration 2023-12-07T23:25:31.494 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-07T23:25:31.575 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T23:25:31.603 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 23:20:29.167448', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk7r/sl5FVd4xb7HmAkal/vir+w88NJs5R3zxla6Lu2/yqjGtBg3JQ18Zye8EGgSE2uX+NVcIwyM+LRpgdP3FM='} 2023-12-07T23:25:31.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 23:20:29.165557', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA+p96ecYG4L+/ocaIhrFJzRHy5EQHa+ckbt4N23xqMZTSH7vVmpyFuMTVTnlnXutsIT1XFEzStsPoiZsgnm2k='} 2023-12-07T23:25:31.626 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-07T23:25:31.664 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-07T23:25:31.665 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-07T23:25:31.665 INFO:teuthology.run_tasks:Running task console_log... 2023-12-07T23:25:31.798 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f93b4821ee0>, signals=[15]) 2023-12-07T23:25:31.799 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-07T23:25:31.886 INFO:teuthology.task.internal:Opening connections... 2023-12-07T23:25:31.887 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2023-12-07T23:25:31.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:25:31.975 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2023-12-07T23:25:31.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:25:32.059 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-07T23:25:32.070 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2023-12-07T23:25:32.087 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2023-12-07T23:25:32.088 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2023-12-07T23:25:32.144 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-07T23:25:32.145 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-07T23:25:32.145 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-07T23:25:32.145 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2023-12-07T23:25:32.170 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2023-12-07T23:25:32.188 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2023-12-07T23:25:32.189 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2023-12-07T23:25:32.244 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2023-12-07T23:25:32.244 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2023-12-07T23:25:32.244 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-07T23:25:32.245 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-07T23:25:32.245 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2023-12-07T23:25:32.267 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-07T23:25:32.277 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-07T23:25:32.454 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-07T23:25:32.454 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2023-12-07T23:25:32.457 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2023-12-07T23:25:32.473 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-07T23:25:32.527 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-07T23:25:32.527 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2023-12-07T23:25:32.530 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2023-12-07T23:25:32.620 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-07T23:25:32.678 INFO:teuthology.run_tasks:Running task kernel... 2023-12-07T23:25:32.721 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-07T23:25:32.722 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-07T23:25:32.722 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-07T23:25:32.722 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-07T23:25:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T23:25:32.739 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2023-12-07T23:25:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T23:25:32.740 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2023-12-07T23:25:32.796 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2023-12-07T23:25:32.796 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2023-12-07T23:25:32.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2023-12-07T23:25:32.797 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-361.el9.x86_64 2023-12-07T23:25:32.797 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-361.el9.x86_64 2023-12-07T23:25:32.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2023-12-07T23:25:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-07T23:25:35.016 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-07T23:25:36.233 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2023-12-07T23:25:36.942 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 2.3 MB/s | 2.0 MB 00:00 2023-12-07T23:25:38.790 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 7.9 MB 00:02 2023-12-07T23:25:39.490 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 7.9 MB 00:02 2023-12-07T23:25:44.608 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 18 MB 00:04 2023-12-07T23:25:45.885 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 18 MB 00:06 2023-12-07T23:25:48.550 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 5.8 MB 00:00 2023-12-07T23:25:50.334 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2023-12-07T23:25:51.499 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 5.8 MB 00:02 2023-12-07T23:25:53.074 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2023-12-07T23:25:54.176 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 20 MB/s | 20 MB 00:00 2023-12-07T23:25:58.063 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 20 MB 00:07 2023-12-07T23:25:58.992 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-12-07T23:26:01.241 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-07T23:26:01.241 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-07T23:26:01.329 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-07T23:26:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 96 M 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 121 M 2023-12-07T23:26:01.341 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:26:02.734 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2023-12-07T23:26:02.793 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 4.3 MB/s | 5.6 MB 00:01 2023-12-07T23:26:04.244 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 7.4 MB/s | 20 MB 00:02 2023-12-07T23:26:04.933 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-07T23:26:04.933 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-07T23:26:05.025 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:26:05.034 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:26:05.035 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:26:05.035 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2023-12-07T23:26:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:26:05.035 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 96 M 2023-12-07T23:26:05.035 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 121 M 2023-12-07T23:26:05.035 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:26:05.044 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 11 MB/s | 38 MB 00:03 2023-12-07T23:26:05.419 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 12 MB/s | 32 MB 00:02 2023-12-07T23:26:05.420 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:26:05.420 INFO:teuthology.orchestra.run.smithi184.stdout:Total 24 MB/s | 96 MB 00:04 2023-12-07T23:26:06.036 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 7.8 MB/s | 5.6 MB 00:00 2023-12-07T23:26:06.278 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:26:06.631 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:26:06.631 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:26:06.878 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 13 MB/s | 20 MB 00:01 2023-12-07T23:26:07.930 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:26:07.930 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:26:08.045 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 14 MB/s | 38 MB 00:02 2023-12-07T23:26:08.436 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 13 MB/s | 32 MB 00:02 2023-12-07T23:26:08.437 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:26:08.437 INFO:teuthology.orchestra.run.smithi129.stdout:Total 28 MB/s | 96 MB 00:03 2023-12-07T23:26:09.293 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:26:09.673 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:26:09.674 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:26:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:26:10.989 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:26:10.990 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:26:11.188 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-07T23:26:11.328 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T23:26:13.638 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:26:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T23:26:14.266 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T23:26:14.323 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-07T23:26:14.453 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T23:26:17.231 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T23:26:17.463 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T23:26:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T23:26:19.024 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:22.079 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T23:26:22.195 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:26.845 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:52.795 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:52.796 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-07T23:26:52.796 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T23:26:52.796 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T23:26:53.240 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:53.240 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:26:53.240 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:26:53.240 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-07T23:26:53.240 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-07T23:26:53.240 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-07T23:26:53.241 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-07T23:26:53.241 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:26:53.241 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:26:53.738 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T23:26:53.887 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T23:26:53.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T23:26:53.887 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-07T23:26:53.888 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-07T23:26:53.888 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-07T23:26:53.888 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-07T23:26:53.888 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2023-12-07T23:26:53.888 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T23:26:54.005 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T23:26:54.005 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T23:26:54.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2023-12-07T23:26:54.042 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 68.el9 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8264254 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-07T23:26:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:26:54.044 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2023-12-07T23:26:54.045 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-07T23:26:54.045 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-390.el9.x86_64 2023-12-07T23:26:54.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-07T23:26:54.790 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2023-12-07T23:26:55.776 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:55.818 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:56.201 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-07T23:26:56.243 INFO:teuthology.orchestra.run.smithi184.stderr:done 2023-12-07T23:26:56.245 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2023-12-07T23:26:56.261 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.TIVVscKJt0 2023-12-07T23:26:56.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TIVVscKJt0 2023-12-07T23:26:56.296 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:56.296 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-07T23:26:56.296 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T23:26:56.296 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T23:26:56.328 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.TIVVscKJt0 2023-12-07T23:26:56.471 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.TIVVscKJt0 is 6KB 2023-12-07T23:26:56.520 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.TIVVscKJt0 2023-12-07T23:26:56.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2023-12-07T23:26:56.607 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-07T23:26:56.607 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-07T23:26:56.607 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-07T23:26:56.607 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-07T23:26:56.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-07T23:26:56.666 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T23:26:56.666 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:26:56.666 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:26:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-07T23:26:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-07T23:26:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-07T23:26:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-07T23:26:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:26:56.667 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:26:56.748 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2023-12-07T23:26:57.184 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T23:26:57.327 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T23:26:57.327 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T23:26:57.327 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-07T23:26:57.327 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-07T23:26:57.327 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-07T23:26:57.327 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-07T23:26:57.327 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2023-12-07T23:26:57.327 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T23:26:57.446 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T23:26:57.446 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T23:26:57.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2023-12-07T23:26:57.483 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2023-12-07T23:26:57.484 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2023-12-07T23:26:57.485 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:26:57.485 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-07T23:26:57.485 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-07T23:26:57.485 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-07T23:26:57.485 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2023-12-07T23:26:57.485 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:26:57.485 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2023-12-07T23:26:57.486 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-07T23:26:57.486 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-390.el9.x86_64 2023-12-07T23:26:57.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-07T23:26:58.143 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2023-12-07T23:26:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-07T23:26:59.596 INFO:teuthology.orchestra.run.smithi129.stderr:done 2023-12-07T23:26:59.599 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2023-12-07T23:26:59.655 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.Nww1eCbjW6 2023-12-07T23:26:59.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Nww1eCbjW6 2023-12-07T23:26:59.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.Nww1eCbjW6 2023-12-07T23:26:59.861 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.Nww1eCbjW6 is 6KB 2023-12-07T23:26:59.909 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.Nww1eCbjW6 2023-12-07T23:26:59.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2023-12-07T23:26:59.988 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-07T23:26:59.988 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-07T23:26:59.988 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-07T23:26:59.988 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-07T23:26:59.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-07T23:27:00.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2023-12-07T23:27:26.778 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-07T23:27:26.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-12-07T23:27:26.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:27:30.142 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-07T23:27:30.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-12-07T23:27:30.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:27:45.251 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-12-07T23:27:48.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-12-07T23:27:54.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-12-07T23:27:54.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:27:57.314 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-12-07T23:27:57.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-12-07T23:27:57.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:28:00.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-12-07T23:28:09.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-12-07T23:28:09.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:28:12.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-12-07T23:28:12.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:28:41.078 DEBUG:teuthology.orchestra.run.smithi184:> true 2023-12-07T23:28:41.565 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-12-07T23:28:41.565 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-390.el9.x86_64"... 2023-12-07T23:28:41.566 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2023-12-07T23:28:41.591 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-390.el9.x86_64 2023-12-07T23:28:41.591 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-07T23:28:41.591 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-07T23:28:41.591 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-07T23:28:42.593 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-07T23:28:42.593 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-07T23:28:42.645 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2023-12-07T23:28:42.827 DEBUG:teuthology.parallel:result is None 2023-12-07T23:28:44.524 DEBUG:teuthology.orchestra.run.smithi129:> true 2023-12-07T23:28:44.920 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-12-07T23:28:44.920 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-390.el9.x86_64"... 2023-12-07T23:28:44.920 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2023-12-07T23:28:44.935 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-390.el9.x86_64 2023-12-07T23:28:44.935 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-07T23:28:44.935 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-07T23:28:44.935 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-07T23:28:45.936 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-07T23:28:45.937 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-07T23:28:46.047 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2023-12-07T23:28:46.073 DEBUG:teuthology.parallel:result is None 2023-12-07T23:28:46.074 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-07T23:28:46.083 INFO:teuthology.task.internal:Creating test directory... 2023-12-07T23:28:46.084 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-07T23:28:46.086 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-07T23:28:46.108 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-07T23:28:46.127 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-07T23:28:46.160 INFO:teuthology.task.internal:Creating archive directory... 2023-12-07T23:28:46.160 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-07T23:28:46.162 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-07T23:28:46.288 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-07T23:28:46.296 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-07T23:28:46.296 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-07T23:28:46.299 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-07T23:28:46.335 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T23:28:46.350 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T23:28:46.356 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T23:28:46.383 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T23:28:46.387 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-07T23:28:46.395 INFO:teuthology.task.internal:Configuring sudo... 2023-12-07T23:28:46.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-07T23:28:46.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-07T23:28:46.438 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-07T23:28:46.450 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-07T23:28:46.451 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-07T23:28:46.482 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-07T23:28:46.506 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T23:28:46.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T23:28:46.667 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T23:28:46.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T23:28:46.764 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:28:46.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-07T23:28:46.832 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T23:28:46.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T23:28:46.956 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T23:28:47.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T23:28:47.047 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:28:47.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-07T23:28:47.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2023-12-07T23:28:47.122 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2023-12-07T23:28:47.183 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-07T23:28:47.212 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-07T23:28:47.686 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-07T23:28:47.693 INFO:teuthology.task.internal:Starting timer... 2023-12-07T23:28:47.693 INFO:teuthology.run_tasks:Running task pcp... 2023-12-07T23:28:47.739 INFO:teuthology.run_tasks:Running task selinux... 2023-12-07T23:28:47.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2023-12-07T23:28:47.829 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2023-12-07T23:28:47.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2023-12-07T23:28:47.895 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2023-12-07T23:28:47.897 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-07T23:28:47.898 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2023-12-07T23:28:47.915 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2023-12-07T23:28:47.916 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2023-12-07T23:28:47.951 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2023-12-07T23:28:47.951 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2023-12-07T23:28:47.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-07T23:28:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T23:28:47.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-07T23:28:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T23:28:48.022 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-07T23:28:48.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2023-12-07T23:28:48.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2023-12-07T23:28:48.094 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-07T23:28:48.104 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-07T23:28:48.105 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-07T23:28:48.122 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-07T23:28:48.123 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-07T23:34:23.810 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2023-12-07T23:34:23.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-12-07T23:34:23.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:34:23.897 DEBUG:teuthology.orchestra.run.smithi129:> true 2023-12-07T23:34:23.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-12-07T23:34:23.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-12-07T23:34:23.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T23:34:24.049 DEBUG:teuthology.orchestra.run.smithi184:> true 2023-12-07T23:34:24.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-12-07T23:34:24.117 INFO:teuthology.run_tasks:Running task clock... 2023-12-07T23:34:24.129 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-07T23:34:24.130 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-07T23:34:24.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T23:34:24.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-07T23:34:24.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T23:34:24.161 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-07T23:34:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-07T23:34:24.188 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-07T23:34:24.205 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-07T23:34:24.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2023-12-07T23:34:24.221 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2023-12-07T23:34:24.235 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2023-12-07T23:34:24.238 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-07T23:34:24.248 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2023-12-07T23:34:24.254 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-07T23:34:24.265 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-07T23:34:24.281 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-07T23:34:24.299 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2023-12-07T23:34:24.302 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-07T23:34:24.302 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2023-12-07T23:34:24.302 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.302 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.302 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.302 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.323 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2023-12-07T23:34:24.326 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-07T23:34:24.327 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2023-12-07T23:34:24.327 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.327 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.327 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.327 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T23:34:24.328 INFO:teuthology.run_tasks:Running task install... 2023-12-07T23:34:24.339 DEBUG:teuthology.task.install:project ceph 2023-12-07T23:34:24.339 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'}} 2023-12-07T23:34:24.339 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'} 2023-12-07T23:34:24.339 INFO:teuthology.task.install:Using flavor: default 2023-12-07T23:34:24.345 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-12-07T23:34:24.346 INFO:teuthology.task.install:extra packages: [] 2023-12-07T23:34:24.346 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'tag': None, 'wait_for_package': False} 2023-12-07T23:34:24.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-07T23:34:24.349 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'tag': None, 'wait_for_package': False} 2023-12-07T23:34:24.349 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-07T23:34:24.584 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/ 2023-12-07T23:34:24.584 INFO:teuthology.task.install.rpm:Package version is 19.0.0-83.gc56ad3ea 2023-12-07T23:34:24.594 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/ 2023-12-07T23:34:24.595 INFO:teuthology.task.install.rpm:Package version is 19.0.0-83.gc56ad3ea 2023-12-07T23:34:24.698 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-07T23:34:24.698 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:34:24.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-07T23:34:24.727 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-07T23:34:24.728 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-07T23:34:24.774 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-12-07-0727/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-07T23:34:24.774 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:34:24.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-07T23:34:24.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-07T23:34:24.802 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-07T23:34:24.803 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c56ad3eaf0784383bf22c04e35d26e15c4baeafe/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-07T23:34:24.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-07T23:34:24.876 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-07T23:34:24.943 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2023-12-07T23:34:24.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2023-12-07T23:34:24.950 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-07T23:34:25.017 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2023-12-07T23:34:25.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2023-12-07T23:34:25.186 INFO:teuthology.orchestra.run.smithi184.stdout:73 files removed 2023-12-07T23:34:25.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw 2023-12-07T23:34:25.265 INFO:teuthology.orchestra.run.smithi129.stdout:73 files removed 2023-12-07T23:34:25.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw 2023-12-07T23:34:25.947 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 191 kB/s | 79 kB 00:00 2023-12-07T23:34:26.000 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 204 kB/s | 79 kB 00:00 2023-12-07T23:34:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 30 kB/s | 9.5 kB 00:00 2023-12-07T23:34:26.398 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 24 kB/s | 9.5 kB 00:00 2023-12-07T23:34:26.726 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2023-12-07T23:34:26.765 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2023-12-07T23:34:27.195 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-07T23:34:27.268 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-07T23:34:28.587 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2023-12-07T23:34:28.619 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2023-12-07T23:34:29.453 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 7.9 MB 00:00 2023-12-07T23:34:30.403 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-12-07T23:34:32.853 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 13 MB/s | 18 MB 00:01 2023-12-07T23:34:35.154 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 18 MB 00:04 2023-12-07T23:34:38.145 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.8 MB 00:01 2023-12-07T23:34:39.363 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 5.8 MB 00:00 2023-12-07T23:34:39.471 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 15 kB 00:00 2023-12-07T23:34:40.855 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2023-12-07T23:34:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2023-12-07T23:34:43.783 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 20 MB 00:04 2023-12-07T23:34:47.657 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2023-12-07T23:34:48.505 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-07T23:34:50.241 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:34:50.251 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:34:50.251 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:34:50.251 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:34:50.251 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:34:50.251 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 8.6 M 2023-12-07T23:34:50.251 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2023-12-07T23:34:50.251 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.4 M 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.2 M 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 24 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 743 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-07T23:34:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 499 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.8 M 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 129 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 162 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 321 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 298 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 99 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-07T23:34:50.253 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:34:50.254 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:34:50.254 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:34:50.254 INFO:teuthology.orchestra.run.smithi129.stdout:Install 27 Packages 2023-12-07T23:34:50.254 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2023-12-07T23:34:50.254 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:34:50.254 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 55 M 2023-12-07T23:34:50.254 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:34:50.983 INFO:teuthology.orchestra.run.smithi129.stdout:(1/29): ceph-base-19.0.0-83.gc56ad3ea.el9.x86_6 7.7 MB/s | 5.2 MB 00:00 2023-12-07T23:34:51.042 INFO:teuthology.orchestra.run.smithi129.stdout:(2/29): ceph-selinux-19.0.0-83.gc56ad3ea.el9.x8 413 kB/s | 24 kB 00:00 2023-12-07T23:34:51.088 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:34:51.091 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 8.6 M 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.4 M 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.2 M 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 24 k 2023-12-07T23:34:51.092 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 743 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 499 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.8 M 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 129 k 2023-12-07T23:34:51.093 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 162 k 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 321 k 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 298 k 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 99 k 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout:Install 27 Packages 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 2 Packages 2023-12-07T23:34:51.094 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:34:51.095 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 55 M 2023-12-07T23:34:51.095 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:34:51.159 INFO:teuthology.orchestra.run.smithi129.stdout:(3/29): libcephfs2-19.0.0-83.gc56ad3ea.el9.x86_ 6.2 MB/s | 743 kB 00:00 2023-12-07T23:34:51.242 INFO:teuthology.orchestra.run.smithi129.stdout:(4/29): libradosstriper1-19.0.0-83.gc56ad3ea.el 5.9 MB/s | 499 kB 00:00 2023-12-07T23:34:51.892 INFO:teuthology.orchestra.run.smithi129.stdout:(5/29): ceph-common-19.0.0-83.gc56ad3ea.el9.x86 12 MB/s | 19 MB 00:01 2023-12-07T23:34:52.059 INFO:teuthology.orchestra.run.smithi129.stdout:(6/29): librgw2-19.0.0-83.gc56ad3ea.el9.x86_64. 5.8 MB/s | 4.8 MB 00:00 2023-12-07T23:34:52.126 INFO:teuthology.orchestra.run.smithi129.stdout:(7/29): python3-ceph-argparse-19.0.0-83.gc56ad3 193 kB/s | 45 kB 00:00 2023-12-07T23:34:52.151 INFO:teuthology.orchestra.run.smithi129.stdout:(8/29): python3-ceph-common-19.0.0-83.gc56ad3ea 1.4 MB/s | 129 kB 00:00 2023-12-07T23:34:52.210 INFO:teuthology.orchestra.run.smithi129.stdout:(9/29): python3-cephfs-19.0.0-83.gc56ad3ea.el9. 1.9 MB/s | 162 kB 00:00 2023-12-07T23:34:52.244 INFO:teuthology.orchestra.run.smithi129.stdout:(10/29): python3-rados-19.0.0-83.gc56ad3ea.el9. 3.4 MB/s | 321 kB 00:00 2023-12-07T23:34:52.319 INFO:teuthology.orchestra.run.smithi129.stdout:(11/29): python3-rbd-19.0.0-83.gc56ad3ea.el9.x8 2.7 MB/s | 298 kB 00:00 2023-12-07T23:34:52.344 INFO:teuthology.orchestra.run.smithi129.stdout:(12/29): python3-rgw-19.0.0-83.gc56ad3ea.el9.x8 991 kB/s | 99 kB 00:00 2023-12-07T23:34:52.461 INFO:teuthology.orchestra.run.smithi129.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2023-12-07T23:34:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:(1/29): ceph-base-19.0.0-83.gc56ad3ea.el9.x86_6 5.1 MB/s | 5.2 MB 00:01 2023-12-07T23:34:52.486 INFO:teuthology.orchestra.run.smithi129.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2023-12-07T23:34:52.512 INFO:teuthology.orchestra.run.smithi129.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2023-12-07T23:34:52.531 INFO:teuthology.orchestra.run.smithi184.stdout:(2/29): ceph-selinux-19.0.0-83.gc56ad3ea.el9.x8 483 kB/s | 24 kB 00:00 2023-12-07T23:34:52.562 INFO:teuthology.orchestra.run.smithi129.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-12-07T23:34:52.587 INFO:teuthology.orchestra.run.smithi129.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-12-07T23:34:52.612 INFO:teuthology.orchestra.run.smithi129.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2023-12-07T23:34:52.640 INFO:teuthology.orchestra.run.smithi184.stdout:(3/29): libcephfs2-19.0.0-83.gc56ad3ea.el9.x86_ 6.7 MB/s | 743 kB 00:00 2023-12-07T23:34:52.740 INFO:teuthology.orchestra.run.smithi184.stdout:(4/29): libradosstriper1-19.0.0-83.gc56ad3ea.el 4.9 MB/s | 499 kB 00:00 2023-12-07T23:34:52.829 INFO:teuthology.orchestra.run.smithi129.stdout:(19/29): ceph-radosgw-19.0.0-83.gc56ad3ea.el9.x 3.4 MB/s | 8.6 MB 00:02 2023-12-07T23:34:52.832 INFO:teuthology.orchestra.run.smithi184.stdout:(5/29): ceph-radosgw-19.0.0-83.gc56ad3ea.el9.x8 6.3 MB/s | 8.6 MB 00:01 2023-12-07T23:34:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:(6/29): python3-ceph-argparse-19.0.0-83.gc56ad3 677 kB/s | 45 kB 00:00 2023-12-07T23:34:52.963 INFO:teuthology.orchestra.run.smithi129.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 822 kB/s | 309 kB 00:00 2023-12-07T23:34:52.966 INFO:teuthology.orchestra.run.smithi184.stdout:(7/29): python3-ceph-common-19.0.0-83.gc56ad3ea 1.9 MB/s | 129 kB 00:00 2023-12-07T23:34:53.013 INFO:teuthology.orchestra.run.smithi129.stdout:(21/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2023-12-07T23:34:53.038 INFO:teuthology.orchestra.run.smithi129.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2023-12-07T23:34:53.049 INFO:teuthology.orchestra.run.smithi184.stdout:(8/29): librgw2-19.0.0-83.gc56ad3ea.el9.x86_64. 15 MB/s | 4.8 MB 00:00 2023-12-07T23:34:53.083 INFO:teuthology.orchestra.run.smithi184.stdout:(9/29): python3-cephfs-19.0.0-83.gc56ad3ea.el9. 1.4 MB/s | 162 kB 00:00 2023-12-07T23:34:53.105 INFO:teuthology.orchestra.run.smithi129.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2023-12-07T23:34:53.117 INFO:teuthology.orchestra.run.smithi184.stdout:(10/29): python3-rados-19.0.0-83.gc56ad3ea.el9. 4.7 MB/s | 321 kB 00:00 2023-12-07T23:34:53.150 INFO:teuthology.orchestra.run.smithi184.stdout:(11/29): python3-rbd-19.0.0-83.gc56ad3ea.el9.x8 4.3 MB/s | 298 kB 00:00 2023-12-07T23:34:53.166 INFO:teuthology.orchestra.run.smithi129.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.4 MB/s | 838 kB 00:00 2023-12-07T23:34:53.176 INFO:teuthology.orchestra.run.smithi184.stdout:(12/29): python3-rgw-19.0.0-83.gc56ad3ea.el9.x8 1.7 MB/s | 99 kB 00:00 2023-12-07T23:34:53.200 INFO:teuthology.orchestra.run.smithi129.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2023-12-07T23:34:53.293 INFO:teuthology.orchestra.run.smithi184.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2023-12-07T23:34:53.326 INFO:teuthology.orchestra.run.smithi184.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2023-12-07T23:34:53.333 INFO:teuthology.orchestra.run.smithi129.stdout:(26/29): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2023-12-07T23:34:53.348 INFO:teuthology.orchestra.run.smithi184.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 800 kB/s | 44 kB 00:00 2023-12-07T23:34:53.367 INFO:teuthology.orchestra.run.smithi129.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2023-12-07T23:34:53.399 INFO:teuthology.orchestra.run.smithi184.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.0 MB/s | 661 kB 00:00 2023-12-07T23:34:53.432 INFO:teuthology.orchestra.run.smithi184.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2023-12-07T23:34:53.457 INFO:teuthology.orchestra.run.smithi184.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 708 kB/s | 41 kB 00:00 2023-12-07T23:34:53.509 INFO:teuthology.orchestra.run.smithi129.stdout:(28/29): librados2-19.0.0-83.gc56ad3ea.el9.x86_ 11 MB/s | 3.4 MB 00:00 2023-12-07T23:34:53.591 INFO:teuthology.orchestra.run.smithi184.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2023-12-07T23:34:53.633 INFO:teuthology.orchestra.run.smithi184.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2023-12-07T23:34:53.675 INFO:teuthology.orchestra.run.smithi184.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2023-12-07T23:34:53.693 INFO:teuthology.orchestra.run.smithi129.stdout:(29/29): librbd1-19.0.0-83.gc56ad3ea.el9.x86_64 8.7 MB/s | 3.1 MB 00:00 2023-12-07T23:34:53.699 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:34:53.699 INFO:teuthology.orchestra.run.smithi129.stdout:Total 16 MB/s | 55 MB 00:03 2023-12-07T23:34:53.699 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:34:53.725 INFO:teuthology.orchestra.run.smithi184.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-12-07T23:34:53.730 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:34:53.730 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:34:53.800 INFO:teuthology.orchestra.run.smithi184.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2023-12-07T23:34:53.834 INFO:teuthology.orchestra.run.smithi184.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2023-12-07T23:34:53.859 INFO:teuthology.orchestra.run.smithi184.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2023-12-07T23:34:53.935 INFO:teuthology.orchestra.run.smithi184.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-12-07T23:34:54.199 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:34:54.199 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:34:54.218 INFO:teuthology.orchestra.run.smithi184.stdout:(27/29): librados2-19.0.0-83.gc56ad3ea.el9.x86_ 9.3 MB/s | 3.4 MB 00:00 2023-12-07T23:34:54.435 INFO:teuthology.orchestra.run.smithi184.stdout:(28/29): librbd1-19.0.0-83.gc56ad3ea.el9.x86_64 6.3 MB/s | 3.1 MB 00:00 2023-12-07T23:34:54.627 INFO:teuthology.orchestra.run.smithi184.stdout:(29/29): ceph-common-19.0.0-83.gc56ad3ea.el9.x8 6.0 MB/s | 19 MB 00:03 2023-12-07T23:34:54.631 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:34:54.631 INFO:teuthology.orchestra.run.smithi184.stdout:Total 16 MB/s | 55 MB 00:03 2023-12-07T23:34:54.632 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:34:54.664 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:34:54.664 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:34:55.132 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:34:55.132 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:34:55.567 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:34:55.652 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-07T23:34:55.681 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-07T23:34:55.731 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-07T23:34:56.097 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-07T23:34:56.130 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-07T23:34:56.380 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-07T23:34:56.466 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-07T23:34:56.490 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-07T23:34:56.490 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:34:56.552 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-07T23:34:56.578 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-07T23:34:56.589 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-07T23:34:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-07T23:34:56.650 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 9/31 2023-12-07T23:34:56.657 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-07T23:34:56.719 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-07T23:34:56.742 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-07T23:34:56.807 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-07T23:34:56.915 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-07T23:34:56.950 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-07T23:34:56.974 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-07T23:34:56.990 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-07T23:34:57.023 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-07T23:34:57.048 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-07T23:34:57.051 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-07T23:34:57.084 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-07T23:34:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-07T23:34:57.226 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-07T23:34:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-07T23:34:57.319 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-07T23:34:57.348 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-07T23:34:57.386 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-07T23:34:57.393 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 9/31 2023-12-07T23:34:57.414 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-07T23:34:57.461 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-07T23:34:57.485 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-07T23:34:57.488 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-07T23:34:57.519 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 20/31 2023-12-07T23:34:57.533 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-07T23:34:57.555 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-07T23:34:57.640 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-07T23:34:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-07T23:34:57.716 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-07T23:34:57.749 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-07T23:34:57.777 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-07T23:34:57.810 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-07T23:34:57.989 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-07T23:34:58.017 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-07T23:34:58.072 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-07T23:34:58.120 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-07T23:34:58.147 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 24/31 2023-12-07T23:34:58.148 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-07T23:34:58.186 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 25/31 2023-12-07T23:34:58.238 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 19/31 2023-12-07T23:34:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 20/31 2023-12-07T23:34:58.305 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-07T23:34:58.748 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-07T23:34:58.785 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-07T23:34:58.848 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 23/31 2023-12-07T23:34:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 24/31 2023-12-07T23:34:58.953 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 25/31 2023-12-07T23:35:00.304 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-07T23:35:00.349 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-07T23:35:00.911 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-07T23:35:00.944 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-07T23:35:01.029 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-07T23:35:01.030 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-07T23:35:01.030 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-07T23:35:01.030 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:01.062 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-07T23:35:01.123 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-07T23:35:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-07T23:35:01.721 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 26/31 2023-12-07T23:35:01.761 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-07T23:35:02.014 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 27/31 2023-12-07T23:35:02.015 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-07T23:35:02.015 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-07T23:35:02.015 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:02.047 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-07T23:35:10.604 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2023-12-07T23:35:10.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:11.409 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-07T23:35:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-07T23:35:11.434 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:11.434 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-07T23:35:11.434 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T23:35:11.434 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T23:35:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T23:35:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T23:35:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2023-12-07T23:35:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:12.361 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-07T23:35:12.386 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 29/31 2023-12-07T23:35:12.386 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:12.386 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-07T23:35:12.386 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T23:35:12.386 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T23:35:12.386 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:12.388 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T23:35:12.471 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T23:35:12.472 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 8/31 2023-12-07T23:35:12.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 9/31 2023-12-07T23:35:12.631 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-07T23:35:12.631 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-07T23:35:12.631 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 12/31 2023-12-07T23:35:12.631 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 13/31 2023-12-07T23:35:12.631 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-07T23:35:12.632 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-07T23:35:12.633 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-07T23:35:12.633 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 30/31 2023-12-07T23:35:13.570 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 7/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x8 8/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_ 9/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 10/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 11/31 2023-12-07T23:35:13.571 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 12/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 13/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-07T23:35:13.572 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-07T23:35:13.573 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-07T23:35:13.573 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-07T23:35:13.573 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-07T23:35:13.573 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 28/31 2023-12-07T23:35:13.573 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-07T23:35:13.573 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 30/31 2023-12-07T23:35:13.714 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T23:35:13.714 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-07T23:35:13.715 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:13.717 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:14.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-07T23:35:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: re2-1:20211101-3.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:14.514 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:14.626 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:27 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:14.778 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:14.778 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:14.778 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:35:14.778 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 M 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 164 k 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2023-12-07T23:35:14.779 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:14.780 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 46 M 2023-12-07T23:35:14.780 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 199 M 2023-12-07T23:35:14.780 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:35:14.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test 2023-12-07T23:35:14.927 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-07T23:35:14.977 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-07T23:35:15.019 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-07T23:35:15.044 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): libcephsqlite-19.0.0-83.gc56ad3ea.el9.x8 623 kB/s | 164 kB 00:00 2023-12-07T23:35:15.061 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-07T23:35:15.456 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:27 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:15.600 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 45 M 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-07T23:35:15.601 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 164 k 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2023-12-07T23:35:15.602 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:15.603 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 46 M 2023-12-07T23:35:15.603 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 199 M 2023-12-07T23:35:15.603 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:35:15.785 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-12-07T23:35:15.827 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-07T23:35:15.869 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-07T23:35:15.894 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-12-07T23:35:15.944 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): libcephsqlite-19.0.0-83.gc56ad3ea.el9.x8 481 kB/s | 164 kB 00:00 2023-12-07T23:35:19.111 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): ceph-test-19.0.0-83.gc56ad3ea.el9.x86_64 13 MB/s | 45 MB 00:03 2023-12-07T23:35:19.113 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:19.114 INFO:teuthology.orchestra.run.smithi184.stdout:Total 13 MB/s | 46 MB 00:03 2023-12-07T23:35:19.114 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:35:19.133 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:35:19.133 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:35:19.136 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): ceph-test-19.0.0-83.gc56ad3ea.el9.x86_64 10 MB/s | 45 MB 00:04 2023-12-07T23:35:19.139 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:19.139 INFO:teuthology.orchestra.run.smithi129.stdout:Total 11 MB/s | 46 MB 00:04 2023-12-07T23:35:19.140 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:35:19.160 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:35:19.160 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:35:19.512 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:35:19.512 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:35:19.542 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:35:19.543 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:35:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:35:20.107 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:35:20.150 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-07T23:35:20.187 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-07T23:35:20.204 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-07T23:35:20.237 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-07T23:35:20.260 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-07T23:35:20.269 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-07T23:35:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-07T23:35:20.312 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-07T23:35:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-07T23:35:20.331 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-07T23:35:24.619 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-07T23:35:24.646 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/6 2023-12-07T23:35:24.688 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-07T23:35:24.721 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-07T23:35:26.995 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-07T23:35:26.995 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/6 2023-12-07T23:35:26.995 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/6 2023-12-07T23:35:26.995 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-07T23:35:26.995 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-07T23:35:26.995 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-07T23:35:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 6/6 2023-12-07T23:35:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/6 2023-12-07T23:35:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/6 2023-12-07T23:35:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-07T23:35:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-07T23:35:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-07T23:35:27.374 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-07T23:35:27.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:27.375 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:35:27.375 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 jq-1.6-16.el9.x86_64 2023-12-07T23:35:27.375 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-12-07T23:35:27.375 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-12-07T23:35:27.375 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:27.375 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:19.0.0-83.gc56ad3ea.el9.x86_64 jq-1.6-16.el9.x86_64 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:19.0.0-83.gc56ad3ea.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:27.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph 2023-12-07T23:35:27.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph 2023-12-07T23:35:28.259 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:41 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:28.408 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:28.410 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2023-12-07T23:35:28.410 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:35:28.410 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2023-12-07T23:35:28.410 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:35:28.410 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 6.5 k 2023-12-07T23:35:28.410 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:35:28.410 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 2.1 M 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.5 M 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 243 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.4 M 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 18 M 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-07T23:35:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-07T23:35:28.412 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout:Install 36 Packages 2023-12-07T23:35:28.413 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:28.414 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 30 M 2023-12-07T23:35:28.414 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 112 M 2023-12-07T23:35:28.414 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:35:28.437 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:40 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:28.590 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 6.5 k 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 2.1 M 2023-12-07T23:35:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.5 M 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 243 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 4.4 M 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 18 M 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-07T23:35:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-07T23:35:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout:Install 36 Packages 2023-12-07T23:35:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:28.597 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 30 M 2023-12-07T23:35:28.597 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 112 M 2023-12-07T23:35:28.597 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:35:28.857 INFO:teuthology.orchestra.run.smithi129.stdout:(1/36): ceph-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 31 kB/s | 6.5 kB 00:00 2023-12-07T23:35:29.201 INFO:teuthology.orchestra.run.smithi129.stdout:(2/36): ceph-mgr-19.0.0-83.gc56ad3ea.el9.x86_64 2.7 MB/s | 1.5 MB 00:00 2023-12-07T23:35:29.250 INFO:teuthology.orchestra.run.smithi129.stdout:(3/36): ceph-mds-19.0.0-83.gc56ad3ea.el9.x86_64 3.5 MB/s | 2.1 MB 00:00 2023-12-07T23:35:29.350 INFO:teuthology.orchestra.run.smithi129.stdout:(4/36): ceph-mgr-modules-core-19.0.0-83.gc56ad3 2.4 MB/s | 243 kB 00:00 2023-12-07T23:35:29.493 INFO:teuthology.orchestra.run.smithi129.stdout:(5/36): ceph-mon-19.0.0-83.gc56ad3ea.el9.x86_64 7.0 MB/s | 4.4 MB 00:00 2023-12-07T23:35:29.506 INFO:teuthology.orchestra.run.smithi184.stdout:(1/36): ceph-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 23 kB/s | 6.5 kB 00:00 2023-12-07T23:35:29.544 INFO:teuthology.orchestra.run.smithi129.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 971 kB/s | 187 kB 00:00 2023-12-07T23:35:29.569 INFO:teuthology.orchestra.run.smithi129.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2023-12-07T23:35:29.586 INFO:teuthology.orchestra.run.smithi129.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 746 kB/s | 32 kB 00:00 2023-12-07T23:35:29.604 INFO:teuthology.orchestra.run.smithi129.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-07T23:35:29.629 INFO:teuthology.orchestra.run.smithi129.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 916 kB/s | 39 kB 00:00 2023-12-07T23:35:29.654 INFO:teuthology.orchestra.run.smithi129.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-07T23:35:29.704 INFO:teuthology.orchestra.run.smithi129.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-07T23:35:29.730 INFO:teuthology.orchestra.run.smithi129.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2023-12-07T23:35:29.755 INFO:teuthology.orchestra.run.smithi129.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-07T23:35:29.780 INFO:teuthology.orchestra.run.smithi129.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-07T23:35:29.806 INFO:teuthology.orchestra.run.smithi129.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-07T23:35:29.889 INFO:teuthology.orchestra.run.smithi129.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2023-12-07T23:35:29.963 INFO:teuthology.orchestra.run.smithi184.stdout:(2/36): ceph-mgr-19.0.0-83.gc56ad3ea.el9.x86_64 2.0 MB/s | 1.5 MB 00:00 2023-12-07T23:35:30.081 INFO:teuthology.orchestra.run.smithi184.stdout:(3/36): ceph-mds-19.0.0-83.gc56ad3ea.el9.x86_64 2.5 MB/s | 2.1 MB 00:00 2023-12-07T23:35:30.164 INFO:teuthology.orchestra.run.smithi184.stdout:(4/36): ceph-mgr-modules-core-19.0.0-83.gc56ad3 2.9 MB/s | 243 kB 00:00 2023-12-07T23:35:30.273 INFO:teuthology.orchestra.run.smithi184.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-12-07T23:35:30.298 INFO:teuthology.orchestra.run.smithi129.stdout:(18/36): ceph-osd-19.0.0-83.gc56ad3ea.el9.x86_6 16 MB/s | 18 MB 00:01 2023-12-07T23:35:30.315 INFO:teuthology.orchestra.run.smithi184.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-07T23:35:30.348 INFO:teuthology.orchestra.run.smithi184.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2023-12-07T23:35:30.432 INFO:teuthology.orchestra.run.smithi184.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 567 kB/s | 47 kB 00:00 2023-12-07T23:35:30.465 INFO:teuthology.orchestra.run.smithi129.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 75 kB/s | 43 kB 00:00 2023-12-07T23:35:30.465 INFO:teuthology.orchestra.run.smithi184.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-07T23:35:30.507 INFO:teuthology.orchestra.run.smithi184.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-07T23:35:30.574 INFO:teuthology.orchestra.run.smithi184.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-07T23:35:30.608 INFO:teuthology.orchestra.run.smithi184.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-07T23:35:30.624 INFO:teuthology.orchestra.run.smithi129.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 185 kB/s | 151 kB 00:00 2023-12-07T23:35:30.642 INFO:teuthology.orchestra.run.smithi184.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-12-07T23:35:30.675 INFO:teuthology.orchestra.run.smithi184.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-07T23:35:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:(15/36): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-07T23:35:30.799 INFO:teuthology.orchestra.run.smithi129.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2023-12-07T23:35:30.834 INFO:teuthology.orchestra.run.smithi184.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2023-12-07T23:35:30.984 INFO:teuthology.orchestra.run.smithi184.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2023-12-07T23:35:31.024 INFO:teuthology.orchestra.run.smithi129.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 78 kB/s | 18 kB 00:00 2023-12-07T23:35:31.049 INFO:teuthology.orchestra.run.smithi129.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 229 kB/s | 172 kB 00:00 2023-12-07T23:35:31.060 INFO:teuthology.orchestra.run.smithi184.stdout:(18/36): ceph-mon-19.0.0-83.gc56ad3ea.el9.x86_6 2.9 MB/s | 4.4 MB 00:01 2023-12-07T23:35:31.077 INFO:teuthology.orchestra.run.smithi184.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 473 kB/s | 43 kB 00:00 2023-12-07T23:35:31.152 INFO:teuthology.orchestra.run.smithi184.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2023-12-07T23:35:31.186 INFO:teuthology.orchestra.run.smithi184.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 320 kB/s | 11 kB 00:00 2023-12-07T23:35:31.191 INFO:teuthology.orchestra.run.smithi129.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 139 kB/s | 23 kB 00:00 2023-12-07T23:35:31.211 INFO:teuthology.orchestra.run.smithi184.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2023-12-07T23:35:31.216 INFO:teuthology.orchestra.run.smithi129.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 477 kB/s | 358 kB 00:00 2023-12-07T23:35:31.228 INFO:teuthology.orchestra.run.smithi184.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 420 kB/s | 18 kB 00:00 2023-12-07T23:35:31.242 INFO:teuthology.orchestra.run.smithi129.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 101 kB/s | 19 kB 00:00 2023-12-07T23:35:31.245 INFO:teuthology.orchestra.run.smithi184.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 686 kB/s | 23 kB 00:00 2023-12-07T23:35:31.262 INFO:teuthology.orchestra.run.smithi184.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 576 kB/s | 19 kB 00:00 2023-12-07T23:35:31.376 INFO:teuthology.orchestra.run.smithi129.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 107 kB/s | 20 kB 00:00 2023-12-07T23:35:31.402 INFO:teuthology.orchestra.run.smithi129.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 252 kB/s | 46 kB 00:00 2023-12-07T23:35:31.424 INFO:teuthology.orchestra.run.smithi129.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 433 kB/s | 79 kB 00:00 2023-12-07T23:35:31.438 INFO:teuthology.orchestra.run.smithi184.stdout:(26/36): ceph-osd-19.0.0-83.gc56ad3ea.el9.x86_6 12 MB/s | 18 MB 00:01 2023-12-07T23:35:31.454 INFO:teuthology.orchestra.run.smithi184.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 94 kB/s | 20 kB 00:00 2023-12-07T23:35:31.471 INFO:teuthology.orchestra.run.smithi184.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 221 kB/s | 46 kB 00:00 2023-12-07T23:35:31.498 INFO:teuthology.orchestra.run.smithi184.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2023-12-07T23:35:31.513 INFO:teuthology.orchestra.run.smithi184.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2023-12-07T23:35:31.547 INFO:teuthology.orchestra.run.smithi184.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2023-12-07T23:35:31.564 INFO:teuthology.orchestra.run.smithi184.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2023-12-07T23:35:31.591 INFO:teuthology.orchestra.run.smithi129.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 86 kB/s | 16 kB 00:00 2023-12-07T23:35:31.615 INFO:teuthology.orchestra.run.smithi184.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2023-12-07T23:35:31.616 INFO:teuthology.orchestra.run.smithi129.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 469 kB/s | 90 kB 00:00 2023-12-07T23:35:31.640 INFO:teuthology.orchestra.run.smithi184.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2023-12-07T23:35:31.656 INFO:teuthology.orchestra.run.smithi184.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 477 kB/s | 20 kB 00:00 2023-12-07T23:35:31.690 INFO:teuthology.orchestra.run.smithi184.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2023-12-07T23:35:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:Total 9.8 MB/s | 30 MB 00:03 2023-12-07T23:35:31.705 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:35:31.716 INFO:teuthology.orchestra.run.smithi129.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 800 kB/s | 272 kB 00:00 2023-12-07T23:35:31.746 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:35:31.747 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:35:31.775 INFO:teuthology.orchestra.run.smithi129.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 194 kB/s | 36 kB 00:00 2023-12-07T23:35:31.909 INFO:teuthology.orchestra.run.smithi129.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2023-12-07T23:35:31.942 INFO:teuthology.orchestra.run.smithi129.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 705 kB/s | 230 kB 00:00 2023-12-07T23:35:31.967 INFO:teuthology.orchestra.run.smithi129.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 104 kB/s | 20 kB 00:00 2023-12-07T23:35:31.973 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:31.973 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.5 MB/s | 30 MB 00:03 2023-12-07T23:35:31.985 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:35:32.028 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:35:32.028 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:35:32.173 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:35:32.173 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:35:32.471 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:35:32.471 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:35:33.495 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:35:33.551 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-07T23:35:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-07T23:35:33.636 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-07T23:35:33.705 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-07T23:35:33.752 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-07T23:35:33.785 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-07T23:35:33.828 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:35:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-07T23:35:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-07T23:35:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-07T23:35:33.961 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-07T23:35:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-07T23:35:34.004 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-07T23:35:34.055 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-07T23:35:34.094 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-07T23:35:34.105 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-07T23:35:34.138 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-07T23:35:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-07T23:35:34.185 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-07T23:35:34.230 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-07T23:35:34.234 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-07T23:35:34.288 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-07T23:35:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-07T23:35:34.349 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-07T23:35:34.421 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-07T23:35:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-07T23:35:34.475 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-07T23:35:34.513 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-07T23:35:34.514 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-07T23:35:34.544 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-07T23:35:34.563 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-07T23:35:34.579 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-07T23:35:34.609 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-07T23:35:34.618 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-07T23:35:34.684 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-07T23:35:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T23:35:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-07T23:35:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-07T23:35:34.803 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-07T23:35:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-07T23:35:34.860 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-07T23:35:34.862 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-07T23:35:34.899 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-07T23:35:34.941 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-07T23:35:34.949 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-07T23:35:34.972 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-07T23:35:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-07T23:35:35.048 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-07T23:35:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-07T23:35:35.082 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T23:35:35.116 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-07T23:35:35.156 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-07T23:35:35.191 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-07T23:35:35.199 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-07T23:35:35.225 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-07T23:35:35.315 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-07T23:35:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T23:35:35.357 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-07T23:35:35.429 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-07T23:35:35.494 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 31/36 2023-12-07T23:35:35.523 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-07T23:35:35.547 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-07T23:35:35.548 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:35.548 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-07T23:35:35.548 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T23:35:35.548 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T23:35:35.548 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:35.566 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-07T23:35:35.691 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T23:35:35.850 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 31/36 2023-12-07T23:35:35.879 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-07T23:35:35.904 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 32/36 2023-12-07T23:35:35.904 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:35.904 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-07T23:35:35.904 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T23:35:35.904 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T23:35:35.904 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-07T23:35:37.178 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-07T23:35:37.178 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:37.178 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-07T23:35:37.178 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T23:35:37.178 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T23:35:37.179 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:37.421 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-07T23:35:37.447 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 33/36 2023-12-07T23:35:37.447 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:37.447 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-07T23:35:37.448 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T23:35:37.448 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T23:35:37.448 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:37.617 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-07T23:35:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-07T23:35:37.641 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:37.641 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-07T23:35:37.641 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T23:35:37.641 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T23:35:37.641 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:37.843 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-07T23:35:37.865 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-07T23:35:37.865 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-07T23:35:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T23:35:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T23:35:37.866 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:37.889 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-07T23:35:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 34/36 2023-12-07T23:35:37.912 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:37.912 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-07T23:35:37.912 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T23:35:37.912 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T23:35:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:37.925 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-07T23:35:38.125 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-07T23:35:38.146 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 35/36 2023-12-07T23:35:38.146 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:38.146 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-07T23:35:38.146 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T23:35:38.147 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T23:35:38.147 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:38.212 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 6/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-07T23:35:39.833 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-07T23:35:39.834 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-07T23:35:39.834 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-07T23:35:39.834 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-07T23:35:39.834 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-07T23:35:39.835 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-07T23:35:39.835 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-07T23:35:39.835 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-07T23:35:39.836 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-07T23:35:39.836 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-07T23:35:39.836 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-07T23:35:39.836 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-07T23:35:39.836 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T23:35:39.836 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-07T23:35:39.836 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-07T23:35:39.837 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-07T23:35:39.837 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-07T23:35:39.837 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-07T23:35:39.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-07T23:35:39.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-07T23:35:39.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-07T23:35:39.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T23:35:39.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-07T23:35:39.838 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-07T23:35:39.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-07T23:35:39.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-07T23:35:39.839 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-07T23:35:40.090 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 36/36 2023-12-07T23:35:40.090 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/36 2023-12-07T23:35:40.090 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/36 2023-12-07T23:35:40.090 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 3/36 2023-12-07T23:35:40.090 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 4/36 2023-12-07T23:35:40.090 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 5/36 2023-12-07T23:35:40.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.no 6/36 2023-12-07T23:35:40.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-07T23:35:40.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-07T23:35:40.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-07T23:35:40.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-07T23:35:40.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-07T23:35:40.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-07T23:35:40.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-07T23:35:40.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-07T23:35:40.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-07T23:35:40.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-07T23:35:40.094 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-07T23:35:40.094 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-07T23:35:40.094 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-07T23:35:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T23:35:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-07T23:35:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-07T23:35:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-07T23:35:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-07T23:35:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: lua-5.4.4-4.el9.x86_64 2023-12-07T23:35:40.760 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-07T23:35:40.761 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-07T23:35:40.762 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-07T23:35:40.763 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:40.763 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:40.993 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-07T23:35:40.993 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:40.993 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:35:40.993 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-07T23:35:40.994 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-07T23:35:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-07T23:35:40.996 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:40.997 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:41.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base 2023-12-07T23:35:41.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base 2023-12-07T23:35:41.591 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:53 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:41.686 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:35:41.735 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:41.736 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:35:41.736 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:41.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install cephadm 2023-12-07T23:35:41.861 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:54 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:41.955 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:35:42.004 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:42.005 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:35:42.005 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:42.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install cephadm 2023-12-07T23:35:42.375 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:54 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:42.516 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 802 k 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 802 k 2023-12-07T23:35:42.517 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 817 k 2023-12-07T23:35:42.518 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:35:42.648 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:55 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:42.793 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:42.794 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:42.794 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:35:42.794 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 802 k 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 802 k 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 817 k 2023-12-07T23:35:42.795 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:35:42.966 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm-19.0.0-83.gc56ad3ea.el9.noarch.rpm 1.7 MB/s | 802 kB 00:00 2023-12-07T23:35:42.967 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:42.967 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.7 MB/s | 802 kB 00:00 2023-12-07T23:35:42.967 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:35:42.972 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:35:42.972 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:35:42.978 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:35:42.978 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:35:43.053 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:35:43.083 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm-19.0.0-83.gc56ad3ea.el9.noarch.rpm 2.7 MB/s | 802 kB 00:00 2023-12-07T23:35:43.083 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:43.083 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.7 MB/s | 802 kB 00:00 2023-12-07T23:35:43.084 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:35:43.089 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:35:43.089 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:35:43.095 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:35:43.095 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:35:43.169 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:35:43.565 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:43.657 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:43.705 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:44.001 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:44.077 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:44.308 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:44.308 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:44.309 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:35:44.309 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:44.309 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:44.309 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:44.373 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:35:44.373 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:44.373 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:35:44.373 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:44.373 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:44.373 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:44.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache 2023-12-07T23:35:44.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache 2023-12-07T23:35:45.132 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:57 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:45.157 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:58 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:45.281 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 145 k 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 145 k 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 440 k 2023-12-07T23:35:45.282 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:35:45.310 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 145 k 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 145 k 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 440 k 2023-12-07T23:35:45.311 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:35:45.647 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-19.0.0-83.gc56ad3ea 400 kB/s | 145 kB 00:00 2023-12-07T23:35:45.647 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:45.647 INFO:teuthology.orchestra.run.smithi184.stdout:Total 398 kB/s | 145 kB 00:00 2023-12-07T23:35:45.647 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:35:45.655 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:35:45.655 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:35:45.657 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-19.0.0-83.gc56ad3ea 418 kB/s | 145 kB 00:00 2023-12-07T23:35:45.657 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:45.658 INFO:teuthology.orchestra.run.smithi129.stdout:Total 417 kB/s | 145 kB 00:00 2023-12-07T23:35:45.658 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:35:45.665 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:35:45.665 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:35:45.707 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:35:45.707 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:35:45.716 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:35:45.716 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:35:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:35:45.858 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:35:45.922 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-07T23:35:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-07T23:35:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-07T23:35:45.946 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:45.947 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-07T23:35:45.947 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-07T23:35:45.949 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:35:45.949 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-07T23:35:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:46.934 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-07T23:35:46.934 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:46.934 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:35:46.934 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:46.934 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:46.935 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:46.958 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el 1/1 2023-12-07T23:35:46.958 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:46.958 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:35:46.958 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:35:46.958 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:46.958 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:47.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr 2023-12-07T23:35:47.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr 2023-12-07T23:35:47.760 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:59 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:47.785 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:00 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:35:47.879 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-mgr-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:35:47.905 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:47.906 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:35:47.906 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:47.928 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:47.929 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:35:47.929 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:47.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-dashboard 2023-12-07T23:35:47.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-dashboard 2023-12-07T23:35:48.545 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:00 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:48.565 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:01 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:48.690 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:48.690 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:48.690 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 1.8 M 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 28 k 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 15 k 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:35:48.691 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:35:48.692 INFO:teuthology.orchestra.run.smithi184.stdout:Install 7 Packages 2023-12-07T23:35:48.692 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:48.692 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.1 M 2023-12-07T23:35:48.692 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 8.7 M 2023-12-07T23:35:48.692 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:35:48.709 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:48.711 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:48.711 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:35:48.711 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:48.712 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:35:48.712 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 1.8 M 2023-12-07T23:35:48.712 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:35:48.712 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 28 k 2023-12-07T23:35:48.712 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 15 k 2023-12-07T23:35:48.712 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-07T23:35:48.713 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-07T23:35:48.713 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-07T23:35:48.713 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2023-12-07T23:35:48.713 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-07T23:35:48.713 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:48.713 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:35:48.713 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:35:48.714 INFO:teuthology.orchestra.run.smithi129.stdout:Install 7 Packages 2023-12-07T23:35:48.714 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:48.714 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.1 M 2023-12-07T23:35:48.714 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 8.7 M 2023-12-07T23:35:48.714 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:35:48.994 INFO:teuthology.orchestra.run.smithi129.stdout:(1/7): ceph-grafana-dashboards-19.0.0-83.gc56ad 140 kB/s | 28 kB 00:00 2023-12-07T23:35:49.028 INFO:teuthology.orchestra.run.smithi129.stdout:(2/7): ceph-prometheus-alerts-19.0.0-83.gc56ad3 64 kB/s | 15 kB 00:00 2023-12-07T23:35:49.170 INFO:teuthology.orchestra.run.smithi129.stdout:(3/7): ceph-mgr-dashboard-19.0.0-83.gc56ad3ea.e 4.7 MB/s | 1.8 MB 00:00 2023-12-07T23:35:49.187 INFO:teuthology.orchestra.run.smithi129.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2023-12-07T23:35:49.308 INFO:teuthology.orchestra.run.smithi184.stdout:(1/7): ceph-prometheus-alerts-19.0.0-83.gc56ad3 60 kB/s | 15 kB 00:00 2023-12-07T23:35:49.334 INFO:teuthology.orchestra.run.smithi184.stdout:(2/7): ceph-grafana-dashboards-19.0.0-83.gc56ad 102 kB/s | 28 kB 00:00 2023-12-07T23:35:49.337 INFO:teuthology.orchestra.run.smithi129.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 184 kB/s | 31 kB 00:00 2023-12-07T23:35:49.354 INFO:teuthology.orchestra.run.smithi129.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 126 kB/s | 41 kB 00:00 2023-12-07T23:35:49.442 INFO:teuthology.orchestra.run.smithi184.stdout:(3/7): ceph-mgr-dashboard-19.0.0-83.gc56ad3ea.e 4.6 MB/s | 1.8 MB 00:00 2023-12-07T23:35:49.484 INFO:teuthology.orchestra.run.smithi184.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 272 kB/s | 41 kB 00:00 2023-12-07T23:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout:(5/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 45 kB/s | 9.0 kB 00:00 2023-12-07T23:35:49.534 INFO:teuthology.orchestra.run.smithi184.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 339 kB/s | 31 kB 00:00 2023-12-07T23:35:49.571 INFO:teuthology.orchestra.run.smithi129.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 491 kB/s | 188 kB 00:00 2023-12-07T23:35:49.572 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:49.573 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-12-07T23:35:49.573 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:35:49.581 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:35:49.581 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:35:49.601 INFO:teuthology.orchestra.run.smithi184.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2023-12-07T23:35:49.604 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:49.604 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.3 MB/s | 2.1 MB 00:00 2023-12-07T23:35:49.604 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:35:49.612 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:35:49.613 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:35:49.646 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:35:49.647 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:35:49.678 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:35:49.679 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:35:49.942 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:35:49.978 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:35:49.989 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-07T23:35:50.031 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-07T23:35:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-07T23:35:50.077 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-07T23:35:50.086 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-07T23:35:50.110 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-07T23:35:50.139 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 5/7 2023-12-07T23:35:50.141 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-07T23:35:50.174 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-07T23:35:50.203 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 5/7 2023-12-07T23:35:50.386 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 6/7 2023-12-07T23:35:50.416 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-07T23:35:50.450 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 6/7 2023-12-07T23:35:50.488 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-07T23:35:50.974 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-07T23:35:50.974 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 1/7 2023-12-07T23:35:50.974 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2/7 2023-12-07T23:35:50.974 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 3/7 2023-12-07T23:35:50.974 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-07T23:35:50.974 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-07T23:35:50.974 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-07T23:35:51.035 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 7/7 2023-12-07T23:35:51.035 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.no 1/7 2023-12-07T23:35:51.035 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2/7 2023-12-07T23:35:51.035 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noa 3/7 2023-12-07T23:35:51.035 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-07T23:35:51.036 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-07T23:35:51.036 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-07T23:35:51.292 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-07T23:35:51.293 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-07T23:35:51.293 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-07T23:35:51.293 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:51.293 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:35:51.368 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-07T23:35:51.368 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:51.369 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:35:51.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-07T23:35:51.594 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-07T23:35:52.098 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:05 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:35:52.170 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:04 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:35:52.243 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================== 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================== 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 7.4 M 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-07T23:35:52.244 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================== 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2023-12-07T23:35:52.245 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:35:52.246 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 37 M 2023-12-07T23:35:52.246 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 209 M 2023-12-07T23:35:52.246 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:35:52.314 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:35:52.315 INFO:teuthology.orchestra.run.smithi184.stdout:========================================================================================== 2023-12-07T23:35:52.315 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:35:52.315 INFO:teuthology.orchestra.run.smithi184.stdout:========================================================================================== 2023-12-07T23:35:52.315 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:35:52.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 7.4 M 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:35:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:========================================================================================== 2023-12-07T23:35:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:Install 12 Packages 2023-12-07T23:35:52.317 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:35:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 37 M 2023-12-07T23:35:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 209 M 2023-12-07T23:35:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:35:52.453 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2023-12-07T23:35:52.486 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2023-12-07T23:35:52.547 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 582 kB/s | 34 kB 00:00 2023-12-07T23:35:52.592 INFO:teuthology.orchestra.run.smithi184.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2023-12-07T23:35:52.604 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2023-12-07T23:35:52.625 INFO:teuthology.orchestra.run.smithi184.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2023-12-07T23:35:52.667 INFO:teuthology.orchestra.run.smithi184.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 34 kB 00:00 2023-12-07T23:35:52.717 INFO:teuthology.orchestra.run.smithi184.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-12-07T23:35:52.788 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-12-07T23:35:52.854 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2023-12-07T23:35:52.880 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-07T23:35:52.926 INFO:teuthology.orchestra.run.smithi184.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-12-07T23:35:53.001 INFO:teuthology.orchestra.run.smithi184.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2023-12-07T23:35:53.034 INFO:teuthology.orchestra.run.smithi184.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2023-12-07T23:35:53.142 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2023-12-07T23:35:53.235 INFO:teuthology.orchestra.run.smithi184.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-83 9.4 MB/s | 7.4 MB 00:00 2023-12-07T23:35:53.290 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2023-12-07T23:35:53.311 INFO:teuthology.orchestra.run.smithi184.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2023-12-07T23:35:53.314 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-07T23:35:53.439 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2023-12-07T23:35:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2023-12-07T23:35:53.494 INFO:teuthology.orchestra.run.smithi184.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 760 kB/s | 195 kB 00:00 2023-12-07T23:35:53.606 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-8 5.7 MB/s | 7.4 MB 00:01 2023-12-07T23:35:53.609 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:53.610 INFO:teuthology.orchestra.run.smithi129.stdout:Total 27 MB/s | 37 MB 00:01 2023-12-07T23:35:53.623 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:35:53.646 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:35:53.647 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:35:54.033 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:35:54.034 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:35:54.270 INFO:teuthology.orchestra.run.smithi184.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 257 kB/s | 245 kB 00:00 2023-12-07T23:35:54.273 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:35:54.273 INFO:teuthology.orchestra.run.smithi184.stdout:Total 19 MB/s | 37 MB 00:01 2023-12-07T23:35:54.287 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:35:54.313 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:35:54.313 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:35:54.703 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:35:54.703 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:35:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:35:54.823 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-07T23:35:54.852 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-07T23:35:54.948 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-07T23:35:54.990 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-07T23:35:55.165 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T23:35:55.196 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T23:35:55.281 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-07T23:35:55.405 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:35:55.490 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-07T23:35:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-07T23:35:55.623 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-07T23:35:55.668 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-07T23:35:55.853 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T23:35:55.884 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T23:35:55.933 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-07T23:35:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-07T23:35:56.041 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-07T23:35:56.618 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-07T23:35:56.719 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-07T23:35:57.275 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-07T23:35:57.919 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-07T23:35:58.289 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-07T23:35:58.314 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-07T23:35:58.946 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-07T23:35:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-07T23:36:00.804 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-07T23:36:00.804 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 1/12 2023-12-07T23:36:00.804 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-07T23:36:00.805 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-07T23:36:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-07T23:36:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:01.242 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-07T23:36:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:01.243 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:01.418 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 12/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3e 1/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-07T23:36:01.419 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-07T23:36:01.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-rook 2023-12-07T23:36:01.886 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-07T23:36:01.886 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:01.886 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:01.887 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:02.174 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:15 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:02.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-rook 2023-12-07T23:36:02.320 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:02.321 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:36:02.321 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:36:02.321 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 50 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-07T23:36:02.322 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout:Install 13 Packages 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.0 M 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2023-12-07T23:36:02.323 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:02.534 INFO:teuthology.orchestra.run.smithi129.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 18 kB 00:00 2023-12-07T23:36:02.559 INFO:teuthology.orchestra.run.smithi129.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 165 kB/s | 24 kB 00:00 2023-12-07T23:36:02.609 INFO:teuthology.orchestra.run.smithi129.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-12-07T23:36:02.634 INFO:teuthology.orchestra.run.smithi129.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-07T23:36:02.660 INFO:teuthology.orchestra.run.smithi129.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-07T23:36:02.685 INFO:teuthology.orchestra.run.smithi129.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-12-07T23:36:02.710 INFO:teuthology.orchestra.run.smithi129.stdout:(7/13): ceph-mgr-rook-19.0.0-83.gc56ad3ea.el9.n 168 kB/s | 50 kB 00:00 2023-12-07T23:36:02.814 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:14 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:02.835 INFO:teuthology.orchestra.run.smithi129.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2023-12-07T23:36:02.869 INFO:teuthology.orchestra.run.smithi129.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 81 kB/s | 15 kB 00:00 2023-12-07T23:36:02.961 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:02.961 INFO:teuthology.orchestra.run.smithi129.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 965 kB/s | 197 kB 00:00 2023-12-07T23:36:02.962 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:36:02.962 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:36:02.962 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:36:02.962 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:02.962 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 50 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:02.963 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:02.964 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:36:02.964 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2023-12-07T23:36:02.964 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:02.964 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.0 M 2023-12-07T23:36:02.964 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 26 M 2023-12-07T23:36:02.964 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:02.994 INFO:teuthology.orchestra.run.smithi129.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 6.5 MB/s | 1.0 MB 00:00 2023-12-07T23:36:03.020 INFO:teuthology.orchestra.run.smithi129.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2023-12-07T23:36:03.045 INFO:teuthology.orchestra.run.smithi129.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2023-12-07T23:36:03.049 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:03.049 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2023-12-07T23:36:03.049 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:03.062 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:03.062 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:03.157 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:03.157 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:03.176 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 188 kB/s | 24 kB 00:00 2023-12-07T23:36:03.193 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2023-12-07T23:36:03.242 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-07T23:36:03.268 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-07T23:36:03.293 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-07T23:36:03.318 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2023-12-07T23:36:03.345 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): ceph-mgr-rook-19.0.0-83.gc56ad3ea.el9.n 168 kB/s | 50 kB 00:00 2023-12-07T23:36:03.627 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 49 kB/s | 15 kB 00:00 2023-12-07T23:36:03.682 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:03.761 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 69 kB/s | 32 kB 00:00 2023-12-07T23:36:03.874 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-07T23:36:03.930 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-07T23:36:03.944 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 329 kB/s | 197 kB 00:00 2023-12-07T23:36:03.979 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 272 kB/s | 59 kB 00:00 2023-12-07T23:36:03.985 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-07T23:36:04.022 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-07T23:36:04.064 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-07T23:36:04.070 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 717 kB/s | 90 kB 00:00 2023-12-07T23:36:04.157 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-07T23:36:04.262 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2023-12-07T23:36:04.265 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-07T23:36:04.266 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:04.267 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2023-12-07T23:36:04.267 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:04.279 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:04.280 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-07T23:36:04.377 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:04.387 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:04.900 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-07T23:36:05.051 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-07T23:36:05.080 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-07T23:36:05.104 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-07T23:36:05.113 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-07T23:36:05.161 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-07T23:36:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-07T23:36:05.184 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-07T23:36:05.206 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-07T23:36:05.247 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-07T23:36:05.339 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-07T23:36:05.450 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-07T23:36:05.498 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 1/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-07T23:36:05.946 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-07T23:36:05.947 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-07T23:36:05.947 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-07T23:36:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-07T23:36:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-07T23:36:06.314 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-07T23:36:06.315 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:06.315 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:06.315 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:06.315 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-07T23:36:06.315 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-07T23:36:06.315 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-07T23:36:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-07T23:36:06.317 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-07T23:36:06.317 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:06.317 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:06.329 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-07T23:36:06.377 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-07T23:36:06.408 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-07T23:36:06.578 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-cephadm 2023-12-07T23:36:07.161 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:20 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 13/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 1/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-07T23:36:07.202 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-07T23:36:07.203 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-07T23:36:07.203 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-07T23:36:07.308 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:07.308 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:36:07.308 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:36:07.308 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 133 k 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2023-12-07T23:36:07.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:07.310 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 6.8 M 2023-12-07T23:36:07.310 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 33 M 2023-12-07T23:36:07.310 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:07.593 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-07T23:36:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-07T23:36:07.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-07T23:36:07.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-07T23:36:07.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-07T23:36:07.595 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:07.595 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:07.666 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-07T23:36:07.800 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-12-07T23:36:07.846 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-cephadm 2023-12-07T23:36:07.917 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): ceph-mgr-cephadm-19.0.0-83.gc56ad3ea.el9 317 kB/s | 133 kB 00:00 2023-12-07T23:36:08.292 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 123 kB/s | 58 kB 00:00 2023-12-07T23:36:08.317 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 136 kB/s | 51 kB 00:00 2023-12-07T23:36:08.428 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:20 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:08.450 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 700 kB/s | 548 kB 00:00 2023-12-07T23:36:08.454 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:08.454 INFO:teuthology.orchestra.run.smithi129.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2023-12-07T23:36:08.454 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:08.463 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:08.463 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:08.552 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:08.553 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:08.573 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 133 k 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-07T23:36:08.574 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:08.575 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:08.575 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:36:08.575 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2023-12-07T23:36:08.575 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:08.575 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 6.8 M 2023-12-07T23:36:08.575 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 33 M 2023-12-07T23:36:08.575 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:08.822 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-07T23:36:08.848 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-12-07T23:36:09.015 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-07T23:36:09.061 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-07T23:36:09.089 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): ceph-mgr-cephadm-19.0.0-83.gc56ad3ea.el9 306 kB/s | 133 kB 00:00 2023-12-07T23:36:09.574 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-07T23:36:09.655 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-07T23:36:09.717 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-07T23:36:09.742 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-07T23:36:10.081 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 51 kB/s | 58 kB 00:01 2023-12-07T23:36:10.148 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 48 kB/s | 51 kB 00:01 2023-12-07T23:36:10.556 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-07T23:36:10.557 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/6 2023-12-07T23:36:10.557 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-07T23:36:10.557 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-07T23:36:10.557 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-07T23:36:10.557 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-07T23:36:10.590 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 310 kB/s | 548 kB 00:01 2023-12-07T23:36:10.595 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:10.596 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.4 MB/s | 6.8 MB 00:02 2023-12-07T23:36:10.596 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:10.606 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:10.606 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:10.699 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:10.699 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-07T23:36:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:10.880 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:11.000 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:11.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse 2023-12-07T23:36:11.173 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-07T23:36:11.241 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-07T23:36:11.708 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:24 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:11.760 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-07T23:36:11.843 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-07T23:36:11.857 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:11.857 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:11.857 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 884 k 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2023-12-07T23:36:11.858 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:11.859 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 963 k 2023-12-07T23:36:11.859 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.8 M 2023-12-07T23:36:11.859 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:11.913 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-07T23:36:11.954 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-07T23:36:11.990 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 610 kB/s | 79 kB 00:00 2023-12-07T23:36:12.188 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-fuse-19.0.0-83.gc56ad3ea.el9.x86_64 2.6 MB/s | 884 kB 00:00 2023-12-07T23:36:12.190 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:12.190 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.8 MB/s | 963 kB 00:00 2023-12-07T23:36:12.190 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:12.211 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:12.212 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:12.287 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:12.287 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:12.472 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:12.610 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-07T23:36:12.679 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:36:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 6/6 2023-12-07T23:36:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 1/6 2023-12-07T23:36:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-07T23:36:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-07T23:36:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-07T23:36:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-07T23:36:13.056 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-07T23:36:13.057 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:13.057 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:13.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse 2023-12-07T23:36:13.359 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:36:13.359 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-07T23:36:13.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-07T23:36:13.679 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:13.679 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:13.679 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-07T23:36:13.679 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:13.680 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:13.896 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:25 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:13.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-volume 2023-12-07T23:36:14.044 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 884 k 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:14.045 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:14.046 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:14.046 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2023-12-07T23:36:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:14.046 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 963 k 2023-12-07T23:36:14.046 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.8 M 2023-12-07T23:36:14.046 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:14.166 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 655 kB/s | 79 kB 00:00 2023-12-07T23:36:14.358 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): ceph-fuse-19.0.0-83.gc56ad3ea.el9.x86_64 2.8 MB/s | 884 kB 00:00 2023-12-07T23:36:14.359 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:14.360 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.0 MB/s | 963 kB 00:00 2023-12-07T23:36:14.360 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:14.380 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:14.380 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:14.456 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:14.456 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:14.534 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:27 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:14.650 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:14.680 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:14.680 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 262 k 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 262 k 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.2 M 2023-12-07T23:36:14.681 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:14.785 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-07T23:36:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:36:15.062 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-volume-19.0.0-83.gc56ad3ea.el9.noarch.rpm 692 kB/s | 262 kB 00:00 2023-12-07T23:36:15.062 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:15.062 INFO:teuthology.orchestra.run.smithi129.stdout:Total 689 kB/s | 262 kB 00:00 2023-12-07T23:36:15.062 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:15.070 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:15.070 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:15.102 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:15.102 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:15.314 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:15.388 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:36:15.403 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:36:15.403 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:36:15.403 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-07T23:36:15.403 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:36:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-07T23:36:15.864 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-07T23:36:15.864 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:15.864 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:15.864 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-07T23:36:15.864 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:15.865 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:16.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-volume 2023-12-07T23:36:16.436 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:36:16.436 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:16.436 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:16.436 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:16.436 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:16.436 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:16.757 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:28 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:16.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel 2023-12-07T23:36:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:16.907 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume noarch 2:19.0.0-83.gc56ad3ea.el9 ceph-noarch 262 k 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 262 k 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 1.2 M 2023-12-07T23:36:16.908 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:17.313 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-volume-19.0.0-83.gc56ad3ea.el9.noarch.rpm 649 kB/s | 262 kB 00:00 2023-12-07T23:36:17.313 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:17.314 INFO:teuthology.orchestra.run.smithi184.stdout:Total 646 kB/s | 262 kB 00:00 2023-12-07T23:36:17.314 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:17.322 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:17.322 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:17.359 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:17.359 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:17.469 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:30 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:17.576 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:17.615 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 126 k 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:17.616 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:36:17.617 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:17.617 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 126 k 2023-12-07T23:36:17.617 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 456 k 2023-12-07T23:36:17.617 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:36:17.665 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:36:17.665 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:36:17.665 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-07T23:36:17.665 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:18.183 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-19.0.0-83.gc56ad3ea.el9.x86_64.r 223 kB/s | 126 kB 00:00 2023-12-07T23:36:18.184 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:18.184 INFO:teuthology.orchestra.run.smithi129.stdout:Total 223 kB/s | 126 kB 00:00 2023-12-07T23:36:18.184 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:18.189 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:18.190 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:18.246 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:18.246 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:18.402 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:18.498 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:18.704 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 1/1 2023-12-07T23:36:18.704 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:18.704 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:18.704 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume-2:19.0.0-83.gc56ad3ea.el9.noarch 2023-12-07T23:36:18.704 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:18.704 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:18.967 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:19.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel 2023-12-07T23:36:19.277 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:19.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:19.277 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:19.277 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:19.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:19.277 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:19.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2 2023-12-07T23:36:19.637 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:31 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:19.783 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 126 k 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 126 k 2023-12-07T23:36:19.785 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 456 k 2023-12-07T23:36:19.785 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:20.094 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:33 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:20.189 INFO:teuthology.orchestra.run.smithi129.stdout:Package libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:20.239 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:20.240 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:36:20.240 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-19.0.0-83.gc56ad3ea.el9.x86_64.r 266 kB/s | 126 kB 00:00 2023-12-07T23:36:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:20.261 INFO:teuthology.orchestra.run.smithi184.stdout:Total 265 kB/s | 126 kB 00:00 2023-12-07T23:36:20.261 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:20.266 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:20.267 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:20.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs-devel 2023-12-07T23:36:20.322 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:20.322 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:20.912 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:33 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:20.995 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:21.059 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 31 k 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 31 k 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 139 k 2023-12-07T23:36:21.060 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:21.295 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:21.295 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:21.295 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:21.295 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:21.295 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:21.295 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:21.303 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs-devel-19.0.0-83.gc56ad3ea.el9.x86_64. 128 kB/s | 31 kB 00:00 2023-12-07T23:36:21.304 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:21.304 INFO:teuthology.orchestra.run.smithi129.stdout:Total 127 kB/s | 31 kB 00:00 2023-12-07T23:36:21.304 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:21.307 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:21.308 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:21.329 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:21.329 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:21.418 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:21.526 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:21.529 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2 2023-12-07T23:36:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:22.132 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:34 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:22.213 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:22.213 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:22.213 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:22.213 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:22.213 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:22.213 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:22.232 INFO:teuthology.orchestra.run.smithi184.stdout:Package libcephfs2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:22.282 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:22.283 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:36:22.283 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:22.358 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs-devel 2023-12-07T23:36:22.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2 2023-12-07T23:36:22.939 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:34 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:23.012 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:36 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:23.085 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 31 k 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 31 k 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 139 k 2023-12-07T23:36:23.086 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:23.107 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:23.158 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:23.159 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:36:23.160 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:23.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1 2023-12-07T23:36:23.330 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs-devel-19.0.0-83.gc56ad3ea.el9.x86_64. 128 kB/s | 31 kB 00:00 2023-12-07T23:36:23.330 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:23.330 INFO:teuthology.orchestra.run.smithi184.stdout:Total 127 kB/s | 31 kB 00:00 2023-12-07T23:36:23.331 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:23.334 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:23.334 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:23.368 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:23.369 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:23.460 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:23.569 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:23.824 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:36 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:23.920 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:23.968 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:23.974 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:23.975 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:36:23.975 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:24.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados 2023-12-07T23:36:24.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:24.251 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:24.251 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:24.251 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:24.251 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:24.252 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:24.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2 2023-12-07T23:36:24.630 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:37 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:24.725 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:24.775 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:24.776 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:36:24.776 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:24.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw 2023-12-07T23:36:25.017 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:37 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:25.114 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:25.166 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:25.166 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:36:25.167 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:25.236 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1 2023-12-07T23:36:25.435 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:38 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:25.529 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:25.579 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:25.580 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:36:25.580 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:25.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs 2023-12-07T23:36:25.823 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:37 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:25.919 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:25.972 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:25.972 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:36:25.973 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:26.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados 2023-12-07T23:36:26.244 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:39 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:26.339 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:26.389 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:26.390 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:36:26.390 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:26.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd 2023-12-07T23:36:26.631 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:38 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:26.726 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rados-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:26.776 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:26.777 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:36:26.777 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:26.846 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw 2023-12-07T23:36:27.045 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:40 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:27.139 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:27.188 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:27.189 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:36:27.189 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:27.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse 2023-12-07T23:36:27.436 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:39 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:27.532 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rgw-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:27.582 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:27.583 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:36:27.583 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:27.659 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs 2023-12-07T23:36:27.841 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:40 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:27.989 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:27.990 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:27.990 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2023-12-07T23:36:27.990 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:27.990 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:27.990 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 85 k 2023-12-07T23:36:27.991 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:27.991 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:27.991 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:27.991 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:36:27.991 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:27.991 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 85 k 2023-12-07T23:36:27.991 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 231 k 2023-12-07T23:36:27.992 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:28.246 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:40 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:28.307 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 270 kB/s | 85 kB 00:00 2023-12-07T23:36:28.307 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:28.307 INFO:teuthology.orchestra.run.smithi129.stdout:Total 269 kB/s | 85 kB 00:00 2023-12-07T23:36:28.307 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:28.314 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:28.314 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:28.340 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-cephfs-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:28.369 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:28.370 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:28.391 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:28.392 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:36:28.392 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:28.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd 2023-12-07T23:36:28.501 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:28.588 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:28.979 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:29.056 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:41 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:29.151 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:36:29.202 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:29.203 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:36:29.203 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:29.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse 2023-12-07T23:36:29.314 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:29.314 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:29.314 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:29.314 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:29.314 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:29.314 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:29.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror 2023-12-07T23:36:29.856 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:41 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:30.003 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 85 k 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:30.004 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:36:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:30.005 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 85 k 2023-12-07T23:36:30.005 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 231 k 2023-12-07T23:36:30.005 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:30.169 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:43 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:30.216 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 404 kB/s | 85 kB 00:00 2023-12-07T23:36:30.217 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:30.217 INFO:teuthology.orchestra.run.smithi184.stdout:Total 402 kB/s | 85 kB 00:00 2023-12-07T23:36:30.217 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:30.224 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:30.224 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:30.278 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:30.278 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:30.317 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:30.318 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:30.318 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2023-12-07T23:36:30.318 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:30.318 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:30.318 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-07T23:36:30.318 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:30.319 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:30.319 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:30.319 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:36:30.319 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:30.319 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.1 M 2023-12-07T23:36:30.319 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 13 M 2023-12-07T23:36:30.319 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:30.521 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:30.703 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 8.0 MB/s | 3.1 MB 00:00 2023-12-07T23:36:30.704 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:30.704 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2023-12-07T23:36:30.704 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:30.713 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:30.713 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:30.785 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:30.786 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:30.949 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:31.200 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:31.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:31.260 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:31.260 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:31.271 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:31.296 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:31.296 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:36:31.296 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-07T23:36:31.296 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T23:36:31.297 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T23:36:31.297 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:31.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror 2023-12-07T23:36:32.068 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:44 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:32.216 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:32.216 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 3.1 M 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.1 M 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 13 M 2023-12-07T23:36:32.217 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:32.433 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:32.434 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:32.434 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:32.434 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:32.434 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:32.434 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:32.611 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2023-12-07T23:36:32.611 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:32.612 INFO:teuthology.orchestra.run.smithi184.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2023-12-07T23:36:32.612 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:32.621 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:32.621 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:32.651 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd 2023-12-07T23:36:32.695 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:32.696 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:33.187 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:33.211 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T23:36:33.211 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-07T23:36:33.211 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T23:36:33.211 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T23:36:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:33.252 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:46 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:33.400 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 170 k 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 170 k 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 494 k 2023-12-07T23:36:33.401 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:33.624 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 765 kB/s | 170 kB 00:00 2023-12-07T23:36:33.625 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:33.625 INFO:teuthology.orchestra.run.smithi129.stdout:Total 760 kB/s | 170 kB 00:00 2023-12-07T23:36:33.625 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:33.633 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:33.633 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:33.690 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:33.690 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:33.837 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:34.369 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:34.369 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:34.370 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:34.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd 2023-12-07T23:36:34.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:34.700 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:36:34.701 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:34.701 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:36:34.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base-debuginfo 2023-12-07T23:36:35.202 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:47 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:35.349 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:35.349 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:35.349 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2023-12-07T23:36:35.349 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 170 k 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 170 k 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 494 k 2023-12-07T23:36:35.350 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:35.513 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:48 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:36:35.664 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:36:35.664 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:35.664 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 73 M 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 21 M 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2023-12-07T23:36:35.665 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:36:35.666 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 94 M 2023-12-07T23:36:35.666 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 513 M 2023-12-07T23:36:35.666 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:36:35.939 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-19.0.0-83.gc56ad3ea.el9.x86_64.rpm 289 kB/s | 170 kB 00:00 2023-12-07T23:36:35.939 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:35.940 INFO:teuthology.orchestra.run.smithi184.stdout:Total 288 kB/s | 170 kB 00:00 2023-12-07T23:36:35.940 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:35.949 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:35.949 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:36.006 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:36.006 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:36.253 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:36.703 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:37.020 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:36:37.020 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:37.020 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:36:37.020 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:36:37.020 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:37.020 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:36:37.260 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base-debuginfo 2023-12-07T23:36:37.886 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:49 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:36:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 73 M 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 21 M 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:36:38.037 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2023-12-07T23:36:38.038 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:36:38.038 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 94 M 2023-12-07T23:36:38.038 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 513 M 2023-12-07T23:36:38.038 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:36:41.461 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): ceph-debuginfo-19.0.0-83.gc56ad3ea.el9.x 3.7 MB/s | 21 MB 00:05 2023-12-07T23:36:45.150 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): ceph-base-debuginfo-19.0.0-83.gc56ad3ea. 10 MB/s | 73 MB 00:07 2023-12-07T23:36:46.192 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): ceph-debuginfo-19.0.0-83.gc56ad3ea.el9.x 2.6 MB/s | 21 MB 00:08 2023-12-07T23:36:46.194 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:46.194 INFO:teuthology.orchestra.run.smithi184.stdout:Total 12 MB/s | 94 MB 00:08 2023-12-07T23:36:46.194 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:36:46.199 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:36:46.199 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:36:46.812 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:36:46.812 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:36:49.452 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-base-debuginfo-19.0.0-83.gc56ad3ea. 5.3 MB/s | 73 MB 00:13 2023-12-07T23:36:49.454 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:36:49.454 INFO:teuthology.orchestra.run.smithi129.stdout:Total 6.8 MB/s | 94 MB 00:13 2023-12-07T23:36:49.454 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:36:49.459 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:36:49.459 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:36:49.822 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:36:50.062 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:36:50.062 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:36:53.063 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:36:57.399 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-07T23:36:57.475 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:37:00.632 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-07T23:37:00.703 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:37:00.959 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:37:00.959 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-07T23:37:01.237 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:37:01.237 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:37:01.237 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:37:01.237 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:37:01.237 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:37:01.237 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:37:01.237 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:37:01.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-common-debuginfo 2023-12-07T23:37:02.045 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:14 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:37:02.191 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 404 M 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:37:02.192 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:37:02.193 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 404 M 2023-12-07T23:37:02.193 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 1.6 G 2023-12-07T23:37:02.193 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:37:04.144 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:37:04.144 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/2 2023-12-07T23:37:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2/2 2023-12-07T23:37:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:37:04.421 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:37:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:37:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:37:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:37:04.421 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:37:04.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-common-debuginfo 2023-12-07T23:37:05.227 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:18 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:37:05.385 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:37:05.385 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:37:05.385 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:37:05.385 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:37:05.386 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:37:05.386 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 404 M 2023-12-07T23:37:05.386 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:37:05.386 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:37:05.387 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:37:05.387 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:37:05.387 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:37:05.387 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 404 M 2023-12-07T23:37:05.388 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.6 G 2023-12-07T23:37:05.388 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:37:30.427 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-common-debuginfo-19.0.0-83.gc56ad3ea.el9.x 14 MB/s | 404 MB 00:28 2023-12-07T23:37:30.427 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:37:30.428 INFO:teuthology.orchestra.run.smithi184.stdout:Total 14 MB/s | 404 MB 00:28 2023-12-07T23:37:30.428 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:37:30.431 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:37:30.431 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:37:32.826 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:37:32.826 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:37:34.347 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-common-debuginfo-19.0.0-83.gc56ad3ea.el9.x 14 MB/s | 404 MB 00:28 2023-12-07T23:37:34.348 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:37:34.348 INFO:teuthology.orchestra.run.smithi129.stdout:Total 14 MB/s | 404 MB 00:28 2023-12-07T23:37:34.348 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:37:34.352 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:37:34.352 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:37:36.749 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:37:36.750 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:38:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:38:13.586 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:38:17.449 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:38:17.927 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:38:22.281 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:38:22.564 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:38:22.564 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:22.564 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:38:22.564 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:38:22.564 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:22.564 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:38:22.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-07T23:38:23.401 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:03:35 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:38:23.548 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.4 M 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:====================================================================================== 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 1.4 M 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 6.4 M 2023-12-07T23:38:23.549 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:38:24.032 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-83 3.0 MB/s | 1.4 MB 00:00 2023-12-07T23:38:24.033 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:38:24.033 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.9 MB/s | 1.4 MB 00:00 2023-12-07T23:38:24.033 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:38:24.037 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:38:24.037 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:38:24.078 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:38:24.079 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:38:24.333 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:38:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-07T23:38:24.906 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-07T23:38:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-07T23:38:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:25.191 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:38:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:38:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:25.191 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:38:25.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-07T23:38:25.958 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:03:37 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:38:26.104 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 173 M 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 173 M 2023-12-07T23:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 731 M 2023-12-07T23:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:38:26.217 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:38:26.492 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:38:26.492 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:26.492 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:38:26.492 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:38:26.492 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:26.492 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:38:26.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-07T23:38:27.310 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:03:40 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:38:27.458 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.4 M 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:38:27.459 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2023-12-07T23:38:27.460 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:38:27.460 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:27.460 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 1.4 M 2023-12-07T23:38:27.460 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 6.4 M 2023-12-07T23:38:27.460 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:38:29.006 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-83 944 kB/s | 1.4 MB 00:01 2023-12-07T23:38:29.006 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:38:29.007 INFO:teuthology.orchestra.run.smithi129.stdout:Total 943 kB/s | 1.4 MB 00:01 2023-12-07T23:38:29.007 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:38:29.011 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:38:29.011 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:38:29.055 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:38:29.056 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:38:29.350 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:38:29.446 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-07T23:38:29.868 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-07T23:38:30.142 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc 1/1 2023-12-07T23:38:30.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:30.143 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:38:30.143 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:38:30.143 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:30.143 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:38:30.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-07T23:38:30.902 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:03:43 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:38:31.047 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:38:31.048 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:38:31.048 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:38:31.048 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:38:31.048 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:38:31.048 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 173 M 2023-12-07T23:38:31.048 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:31.049 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:38:31.049 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:38:31.049 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:38:31.049 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:38:31.049 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 173 M 2023-12-07T23:38:31.049 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 731 M 2023-12-07T23:38:31.049 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:38:37.501 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-radosgw-debuginfo-19.0.0-83.gc56ad3ea.el9. 15 MB/s | 173 MB 00:11 2023-12-07T23:38:37.502 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:38:37.502 INFO:teuthology.orchestra.run.smithi184.stdout:Total 15 MB/s | 173 MB 00:11 2023-12-07T23:38:37.502 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:38:37.506 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:38:37.506 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:38:38.549 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:38:38.549 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:38:56.273 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:38:56.364 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-07T23:39:00.556 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-radosgw-debuginfo-19.0.0-83.gc56ad3ea.el9. 5.8 MB/s | 173 MB 00:29 2023-12-07T23:39:00.557 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:39:00.557 INFO:teuthology.orchestra.run.smithi129.stdout:Total 5.8 MB/s | 173 MB 00:29 2023-12-07T23:39:00.558 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:39:00.562 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:39:00.562 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:39:00.961 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-07T23:39:01.242 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-07T23:39:01.242 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:39:01.242 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:39:01.243 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:39:01.243 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:39:01.243 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:39:01.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test-debuginfo 2023-12-07T23:39:01.601 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:39:01.601 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:39:02.057 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:04:14 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:39:02.202 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 676 M 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:39:02.203 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:39:02.204 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:39:02.204 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:39:02.204 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 676 M 2023-12-07T23:39:02.204 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 3.1 G 2023-12-07T23:39:02.204 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:39:19.545 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:39:19.711 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-07T23:39:24.249 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-07T23:39:24.527 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86 1/1 2023-12-07T23:39:24.527 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:39:24.527 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:39:24.527 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:39:24.527 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:39:24.527 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:39:24.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test-debuginfo 2023-12-07T23:39:25.311 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:04:38 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:39:25.456 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:39:25.457 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:39:25.457 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:39:25.457 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:39:25.457 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:39:25.457 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 676 M 2023-12-07T23:39:25.457 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:39:25.457 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:39:25.458 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:39:25.458 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:39:25.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:39:25.458 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 676 M 2023-12-07T23:39:25.458 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 3.1 G 2023-12-07T23:39:25.458 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:40:10.563 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-test-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 15 MB/s | 676 MB 00:45 2023-12-07T23:40:10.564 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:40:10.564 INFO:teuthology.orchestra.run.smithi129.stdout:Total 15 MB/s | 676 MB 00:45 2023-12-07T23:40:10.564 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:40:10.568 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:40:10.569 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:40:11.075 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-test-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 9.8 MB/s | 676 MB 01:08 2023-12-07T23:40:11.075 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:40:11.076 INFO:teuthology.orchestra.run.smithi184.stdout:Total 9.8 MB/s | 676 MB 01:08 2023-12-07T23:40:11.076 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:40:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:40:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:40:14.557 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:40:14.557 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:40:15.054 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:40:15.055 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:41:25.570 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:41:26.117 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:41:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:35.500 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:35.579 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:35.799 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:35.799 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:35.799 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:41:35.799 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:41:35.799 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:35.799 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:41:35.858 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:35.858 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:41:35.859 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:41:35.859 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:41:36.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base-debuginfo 2023-12-07T23:41:36.147 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base-debuginfo 2023-12-07T23:41:36.676 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:06:49 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:41:36.735 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:06:48 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:41:36.772 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:41:36.821 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:41:36.822 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2023-12-07T23:41:36.823 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:41:36.832 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 is already installed. 2023-12-07T23:41:36.883 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:41:36.884 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-12-07T23:41:36.884 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:41:36.903 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-debuginfo 2023-12-07T23:41:36.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-debuginfo 2023-12-07T23:41:37.492 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:06:50 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:41:37.564 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:06:49 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:41:37.637 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:41:37.638 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:37.638 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:41:37.638 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 19 M 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 101 M 2023-12-07T23:41:37.639 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:41:37.709 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:41:37.710 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:41:37.710 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:41:37.710 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:41:37.710 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:41:37.710 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 19 M 2023-12-07T23:41:37.710 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:37.710 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:41:37.711 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:41:37.711 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:41:37.711 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:37.711 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 19 M 2023-12-07T23:41:37.711 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 101 M 2023-12-07T23:41:37.711 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:41:41.293 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-mgr-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 5.2 MB/s | 19 MB 00:03 2023-12-07T23:41:41.294 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:41:41.294 INFO:teuthology.orchestra.run.smithi129.stdout:Total 5.2 MB/s | 19 MB 00:03 2023-12-07T23:41:41.295 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:41:41.302 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:41:41.302 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:41:41.449 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:41:41.449 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:41:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:41:43.725 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:44.676 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-mgr-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 2.7 MB/s | 19 MB 00:06 2023-12-07T23:41:44.677 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:41:44.677 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.7 MB/s | 19 MB 00:06 2023-12-07T23:41:44.677 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:41:44.681 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:41:44.681 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:41:44.719 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:44.832 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:41:44.832 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:41:45.000 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:45.000 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:45.000 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:41:45.000 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:41:45.000 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:45.000 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:41:45.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mds-debuginfo 2023-12-07T23:41:45.795 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:06:58 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:41:45.946 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 46 M 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:41:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:45.948 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 46 M 2023-12-07T23:41:45.948 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 168 M 2023-12-07T23:41:45.948 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:41:47.005 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:41:47.107 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:48.126 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:48.415 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:48.415 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:48.415 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:41:48.415 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:41:48.415 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:48.415 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:41:48.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mds-debuginfo 2023-12-07T23:41:49.200 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:07:01 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:41:49.345 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 46 M 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:41:49.347 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:41:49.348 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:41:49.348 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:49.348 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 46 M 2023-12-07T23:41:49.348 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 168 M 2023-12-07T23:41:49.348 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:41:49.604 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-mds-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 13 MB/s | 46 MB 00:03 2023-12-07T23:41:49.605 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:41:49.606 INFO:teuthology.orchestra.run.smithi129.stdout:Total 13 MB/s | 46 MB 00:03 2023-12-07T23:41:49.606 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:41:49.610 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:41:49.610 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:41:49.917 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:41:49.917 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:41:52.986 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-mds-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 13 MB/s | 46 MB 00:03 2023-12-07T23:41:52.988 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:41:52.988 INFO:teuthology.orchestra.run.smithi184.stdout:Total 13 MB/s | 46 MB 00:03 2023-12-07T23:41:52.988 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:41:52.991 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:41:52.991 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:41:53.297 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:41:53.297 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:41:54.496 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:41:54.594 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:55.881 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:56.158 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:56.158 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:56.158 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:41:56.159 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:41:56.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:56.159 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:41:56.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mon-debuginfo 2023-12-07T23:41:56.946 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:07:09 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:41:57.092 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 111 M 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 111 M 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 506 M 2023-12-07T23:41:57.093 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:41:57.875 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:41:57.983 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:59.379 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:59.662 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:41:59.662 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:59.662 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:41:59.662 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:41:59.663 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:41:59.663 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:41:59.862 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mon-debuginfo 2023-12-07T23:42:00.449 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:07:12 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:42:00.594 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 111 M 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 111 M 2023-12-07T23:42:00.595 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 506 M 2023-12-07T23:42:00.596 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:42:03.006 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-mon-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 19 MB/s | 111 MB 00:05 2023-12-07T23:42:03.007 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:42:03.008 INFO:teuthology.orchestra.run.smithi129.stdout:Total 19 MB/s | 111 MB 00:05 2023-12-07T23:42:03.008 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:42:03.012 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:42:03.012 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:42:03.698 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:42:03.699 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:42:07.337 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-mon-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 17 MB/s | 111 MB 00:06 2023-12-07T23:42:07.338 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:42:07.338 INFO:teuthology.orchestra.run.smithi184.stdout:Total 17 MB/s | 111 MB 00:06 2023-12-07T23:42:07.339 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:42:07.343 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:42:07.343 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:42:08.027 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:42:08.028 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:42:15.332 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:42:15.420 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:42:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:42:18.972 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:42:18.972 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:42:18.972 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:42:18.972 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:42:18.972 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:42:18.972 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:42:19.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-osd-debuginfo 2023-12-07T23:42:19.625 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:42:19.725 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:42:19.766 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:07:32 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:42:19.912 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 311 M 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 311 M 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.3 G 2023-12-07T23:42:19.913 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:42:22.947 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:42:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:42:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:42:23.223 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:42:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:42:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:42:23.224 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:42:23.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-osd-debuginfo 2023-12-07T23:42:24.026 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:07:36 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:42:24.172 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:42:24.173 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:42:24.173 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:42:24.173 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:42:24.173 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:42:24.173 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 311 M 2023-12-07T23:42:24.173 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:42:24.173 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:42:24.174 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:42:24.174 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:42:24.174 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:42:24.174 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 311 M 2023-12-07T23:42:24.174 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 1.3 G 2023-12-07T23:42:24.174 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:42:37.474 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-osd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 18 MB/s | 311 MB 00:17 2023-12-07T23:42:37.475 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:42:37.475 INFO:teuthology.orchestra.run.smithi129.stdout:Total 18 MB/s | 311 MB 00:17 2023-12-07T23:42:37.475 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:42:37.479 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:42:37.479 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:42:39.328 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:42:39.328 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:42:45.502 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-osd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 15 MB/s | 311 MB 00:21 2023-12-07T23:42:45.504 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:42:45.504 INFO:teuthology.orchestra.run.smithi184.stdout:Total 15 MB/s | 311 MB 00:21 2023-12-07T23:42:45.504 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:42:45.507 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:42:45.507 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:42:47.362 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:42:47.362 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:43:11.154 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:11.235 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:19.057 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:19.326 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:43:19.377 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:19.377 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:19.377 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:19.377 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:19.378 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:19.378 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:19.431 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:19.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse-debuginfo 2023-12-07T23:43:20.202 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:08:33 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:43:20.347 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:43:20.347 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:20.347 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:43:20.347 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:20.347 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:43:20.347 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.9 M 2023-12-07T23:43:20.347 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:20.348 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:43:20.348 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:20.348 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:43:20.348 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:20.348 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 5.9 M 2023-12-07T23:43:20.348 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 28 M 2023-12-07T23:43:20.348 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:43:21.340 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 6.0 MB/s | 5.9 MB 00:00 2023-12-07T23:43:21.341 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:21.342 INFO:teuthology.orchestra.run.smithi129.stdout:Total 5.9 MB/s | 5.9 MB 00:00 2023-12-07T23:43:21.342 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:43:21.345 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:43:21.345 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:43:21.413 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:43:21.414 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:43:22.142 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:22.250 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:22.829 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:23.121 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:23.121 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:23.121 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:23.121 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:23.121 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:23.121 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:23.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel-debuginfo 2023-12-07T23:43:23.891 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:08:36 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:43:24.036 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:43:24.036 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:24.036 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:43:24.036 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 108 k 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 108 k 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 440 k 2023-12-07T23:43:24.037 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:43:24.268 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-debuginfo-19.0.0-83.gc56ad3ea.el 470 kB/s | 108 kB 00:00 2023-12-07T23:43:24.269 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:24.269 INFO:teuthology.orchestra.run.smithi129.stdout:Total 467 kB/s | 108 kB 00:00 2023-12-07T23:43:24.269 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:43:24.273 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:43:24.273 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:43:24.306 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:43:24.306 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:43:24.424 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:24.525 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:24.935 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:25.221 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:25.222 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:25.222 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:25.222 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:25.222 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:25.222 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:25.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2-debuginfo 2023-12-07T23:43:25.981 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:08:38 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:43:26.127 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:43:26.128 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:26.128 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:43:26.128 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.3 M 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 5.3 M 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 24 M 2023-12-07T23:43:26.129 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:43:26.806 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs2-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 7.8 MB/s | 5.3 MB 00:00 2023-12-07T23:43:26.807 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:26.807 INFO:teuthology.orchestra.run.smithi129.stdout:Total 7.8 MB/s | 5.3 MB 00:00 2023-12-07T23:43:26.807 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:43:26.810 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:43:26.810 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:43:26.874 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:43:26.875 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:43:27.530 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:27.547 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:27.620 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:43:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:27.824 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:43:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:27.824 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:43:28.055 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse-debuginfo 2023-12-07T23:43:28.136 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:43:28.422 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:43:28.422 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:28.422 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:28.422 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:28.422 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:28.422 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:28.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2-debuginfo 2023-12-07T23:43:28.641 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:08:40 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:43:28.786 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:43:28.787 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:28.787 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:43:28.787 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:28.787 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.9 M 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 5.9 M 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 28 M 2023-12-07T23:43:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:43:29.199 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:08:42 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:43:29.345 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 44 M 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:43:29.346 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:29.347 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 44 M 2023-12-07T23:43:29.347 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 250 M 2023-12-07T23:43:29.347 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:43:30.292 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 3.9 MB/s | 5.9 MB 00:01 2023-12-07T23:43:30.293 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:30.293 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.9 MB/s | 5.9 MB 00:01 2023-12-07T23:43:30.293 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:43:30.297 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:43:30.298 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:43:30.367 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:43:30.368 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:43:31.100 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:43:31.190 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:31.704 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:31.971 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:31.971 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:31.971 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:43:31.971 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:31.972 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:31.972 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:43:32.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel-debuginfo 2023-12-07T23:43:32.758 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:08:44 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:43:32.905 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:43:32.906 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:32.906 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:43:32.906 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:32.906 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:43:32.906 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 108 k 2023-12-07T23:43:32.906 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:43:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:43:32.907 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 108 k 2023-12-07T23:43:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 440 k 2023-12-07T23:43:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:43:33.445 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-debuginfo-19.0.0-83.gc56ad3ea.el 201 kB/s | 108 kB 00:00 2023-12-07T23:43:33.445 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:33.445 INFO:teuthology.orchestra.run.smithi184.stdout:Total 200 kB/s | 108 kB 00:00 2023-12-07T23:43:33.446 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:43:33.449 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:43:33.449 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:43:33.483 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:43:33.483 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:43:33.603 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:43:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:34.138 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:34.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:34.405 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:34.405 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:43:34.405 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:34.405 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:34.405 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:43:34.508 INFO:teuthology.orchestra.run.smithi129.stdout:librados2-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 8.4 MB/s | 44 MB 00:05 2023-12-07T23:43:34.509 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:34.509 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.4 MB/s | 44 MB 00:05 2023-12-07T23:43:34.510 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:43:34.514 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:43:34.514 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:43:34.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2-debuginfo 2023-12-07T23:43:34.810 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:43:34.811 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:43:35.190 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:08:47 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:43:35.333 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:43:35.334 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 5.3 M 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 5.3 M 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 24 M 2023-12-07T23:43:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:43:35.998 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs2-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 8.0 MB/s | 5.3 MB 00:00 2023-12-07T23:43:35.999 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:35.999 INFO:teuthology.orchestra.run.smithi184.stdout:Total 8.0 MB/s | 5.3 MB 00:00 2023-12-07T23:43:35.999 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:43:36.002 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:43:36.002 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:43:36.066 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:43:36.066 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:43:36.726 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:43:36.846 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:43:37.380 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:43:37.675 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:43:37.675 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:37.676 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:43:37.676 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:37.676 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:37.676 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:43:37.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2-debuginfo 2023-12-07T23:43:38.476 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:08:50 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:43:38.621 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:43:38.621 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:38.621 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:43:38.621 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 44 M 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:38.622 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 44 M 2023-12-07T23:43:38.623 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 250 M 2023-12-07T23:43:38.623 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:43:39.793 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:39.889 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:40.926 INFO:teuthology.orchestra.run.smithi184.stdout:librados2-debuginfo-19.0.0-83.gc56ad3ea.el9.x86 19 MB/s | 44 MB 00:02 2023-12-07T23:43:40.926 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:40.927 INFO:teuthology.orchestra.run.smithi184.stdout:Total 19 MB/s | 44 MB 00:02 2023-12-07T23:43:40.927 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:43:40.931 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:43:40.931 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:43:41.228 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:43:41.228 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:43:41.729 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:42.019 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:42.019 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:42.019 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:42.019 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:42.019 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:42.020 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:42.207 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1-debuginfo 2023-12-07T23:43:42.794 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:08:55 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:43:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:43:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:42.939 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:43:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 51 M 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 305 M 2023-12-07T23:43:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:43:46.191 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:43:46.202 INFO:teuthology.orchestra.run.smithi129.stdout:librbd1-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 16 MB/s | 51 MB 00:03 2023-12-07T23:43:46.203 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:46.204 INFO:teuthology.orchestra.run.smithi129.stdout:Total 16 MB/s | 51 MB 00:03 2023-12-07T23:43:46.204 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:43:46.208 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:43:46.208 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:43:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:46.554 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:43:46.554 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:43:48.165 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:48.436 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:48.437 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:43:48.437 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:48.437 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:48.437 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:43:48.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1-debuginfo 2023-12-07T23:43:49.247 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:01 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:43:49.396 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:43:49.396 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:49.396 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:43:49.396 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:49.396 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 51 M 2023-12-07T23:43:49.397 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 305 M 2023-12-07T23:43:49.398 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:43:51.844 INFO:teuthology.orchestra.run.smithi184.stdout:librbd1-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 21 MB/s | 51 MB 00:02 2023-12-07T23:43:51.845 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:51.846 INFO:teuthology.orchestra.run.smithi184.stdout:Total 21 MB/s | 51 MB 00:02 2023-12-07T23:43:51.846 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:43:51.850 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:43:51.850 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:43:52.191 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:43:52.192 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:43:52.485 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:52.575 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:54.810 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:55.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:55.092 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:55.092 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:55.092 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:55.092 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:55.092 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:55.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs-debuginfo 2023-12-07T23:43:55.883 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:09:08 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:43:56.030 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:43:56.030 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 615 k 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 615 k 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.1 M 2023-12-07T23:43:56.031 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:43:56.307 INFO:teuthology.orchestra.run.smithi129.stdout:python3-cephfs-debuginfo-19.0.0-83.gc56ad3ea.el 2.2 MB/s | 615 kB 00:00 2023-12-07T23:43:56.307 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:56.307 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.2 MB/s | 615 kB 00:00 2023-12-07T23:43:56.307 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:43:56.309 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:43:56.309 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:43:56.346 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:43:56.346 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:56.612 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:57.050 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:43:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:57.345 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:57.346 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:57.538 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados-debuginfo 2023-12-07T23:43:58.118 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:43:58.131 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:09:11 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:43:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:43:58.278 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:43:58.278 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:58.278 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:43:58.278 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 770 k 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 770 k 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.7 M 2023-12-07T23:43:58.279 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:43:58.726 INFO:teuthology.orchestra.run.smithi129.stdout:python3-rados-debuginfo-19.0.0-83.gc56ad3ea.el9 1.7 MB/s | 770 kB 00:00 2023-12-07T23:43:58.726 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:43:58.726 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.7 MB/s | 770 kB 00:00 2023-12-07T23:43:58.727 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:43:58.730 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:43:58.730 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:43:58.768 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:43:58.768 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:43:58.946 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:43:59.049 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-07T23:43:59.424 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-07T23:43:59.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-07T23:43:59.708 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:59.708 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:43:59.708 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:43:59.708 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:43:59.708 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:43:59.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd-debuginfo 2023-12-07T23:44:00.426 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:00.489 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:09:13 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:44:00.635 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:44:00.635 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:00.635 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.1 M 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 1.1 M 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 4.3 M 2023-12-07T23:44:00.636 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:44:00.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:00.717 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:00.717 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:00.717 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:00.717 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:00.717 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:00.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs-debuginfo 2023-12-07T23:44:01.120 INFO:teuthology.orchestra.run.smithi129.stdout:python3-rbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x 2.3 MB/s | 1.1 MB 00:00 2023-12-07T23:44:01.121 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:01.121 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.3 MB/s | 1.1 MB 00:00 2023-12-07T23:44:01.121 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:44:01.124 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:44:01.125 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:44:01.164 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:44:01.165 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:44:01.383 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:44:01.477 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:01.569 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:13 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:44:01.714 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:44:01.715 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:44:01.715 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 615 k 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 615 k 2023-12-07T23:44:01.716 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.1 M 2023-12-07T23:44:01.717 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:44:01.859 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:02.097 INFO:teuthology.orchestra.run.smithi184.stdout:python3-cephfs-debuginfo-19.0.0-83.gc56ad3ea.el 1.6 MB/s | 615 kB 00:00 2023-12-07T23:44:02.097 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:02.098 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.6 MB/s | 615 kB 00:00 2023-12-07T23:44:02.098 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:44:02.101 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:44:02.101 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:44:02.138 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:44:02.138 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:44:02.155 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:02.155 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:02.155 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:44:02.155 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:02.155 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:02.155 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:44:02.307 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:44:02.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw-debuginfo 2023-12-07T23:44:02.422 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:44:02.826 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:44:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:09:15 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:44:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x 1/1 2023-12-07T23:44:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:03.099 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:03.100 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:03.100 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:03.108 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 337 k 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:44:03.109 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:03.110 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:44:03.110 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:03.110 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 337 k 2023-12-07T23:44:03.110 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.0 M 2023-12-07T23:44:03.110 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:44:03.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados-debuginfo 2023-12-07T23:44:03.507 INFO:teuthology.orchestra.run.smithi129.stdout:python3-rgw-debuginfo-19.0.0-83.gc56ad3ea.el9.x 848 kB/s | 337 kB 00:00 2023-12-07T23:44:03.508 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:03.508 INFO:teuthology.orchestra.run.smithi129.stdout:Total 846 kB/s | 337 kB 00:00 2023-12-07T23:44:03.508 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:44:03.512 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:44:03.512 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:44:03.546 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:44:03.546 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:44:03.684 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:44:03.781 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:03.884 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:15 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:44:04.029 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:44:04.030 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:04.031 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:44:04.031 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:04.031 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:44:04.031 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 770 k 2023-12-07T23:44:04.031 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:04.031 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:44:04.031 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:04.032 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:44:04.032 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:04.032 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 770 k 2023-12-07T23:44:04.032 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.7 M 2023-12-07T23:44:04.032 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:44:04.205 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:04.325 INFO:teuthology.orchestra.run.smithi184.stdout:python3-rados-debuginfo-19.0.0-83.gc56ad3ea.el9 2.6 MB/s | 770 kB 00:00 2023-12-07T23:44:04.326 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:04.326 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.5 MB/s | 770 kB 00:00 2023-12-07T23:44:04.326 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:44:04.329 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:44:04.329 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:44:04.366 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:44:04.366 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:44:04.490 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:04.490 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:04.491 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:44:04.491 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:04.491 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:04.491 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:44:04.546 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:44:04.657 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-07T23:44:04.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse-debuginfo 2023-12-07T23:44:05.094 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-07T23:44:05.254 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:09:18 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:44:05.374 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x8 1/1 2023-12-07T23:44:05.375 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:05.375 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:05.375 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:05.375 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:05.375 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:05.401 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:44:05.401 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:05.401 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:44:05.401 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:05.401 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 508 k 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 508 k 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.5 M 2023-12-07T23:44:05.402 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:44:05.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd-debuginfo 2023-12-07T23:44:05.819 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 1.2 MB/s | 508 kB 00:00 2023-12-07T23:44:05.820 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:05.820 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.2 MB/s | 508 kB 00:00 2023-12-07T23:44:05.820 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:44:05.823 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:44:05.823 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:44:05.860 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:44:05.860 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:44:06.014 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:44:06.109 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:06.182 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:18 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:44:06.327 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:44:06.327 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:06.327 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.1 M 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:44:06.328 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:06.329 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 1.1 M 2023-12-07T23:44:06.329 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 4.3 M 2023-12-07T23:44:06.329 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:44:06.499 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:06.628 INFO:teuthology.orchestra.run.smithi184.stdout:python3-rbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x 3.8 MB/s | 1.1 MB 00:00 2023-12-07T23:44:06.628 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:06.629 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.7 MB/s | 1.1 MB 00:00 2023-12-07T23:44:06.629 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:44:06.632 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:44:06.632 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:44:06.671 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:44:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:44:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:06.781 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:44:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:06.781 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:44:06.890 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:44:06.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror-debuginfo 2023-12-07T23:44:06.993 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:07.387 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:07.543 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:09:20 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:07.687 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:44:07.688 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:07.688 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:44:07.688 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:07.688 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:44:07.688 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-07T23:44:07.688 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:07.689 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:44:07.689 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:07.689 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:44:07.689 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:07.689 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 51 M 2023-12-07T23:44:07.689 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 296 M 2023-12-07T23:44:07.689 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:44:07.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw-debuginfo 2023-12-07T23:44:08.512 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:20 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:44:08.658 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:44:08.659 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:08.659 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:44:08.659 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:08.659 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:44:08.659 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 337 k 2023-12-07T23:44:08.659 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:08.660 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:44:08.660 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:08.660 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:44:08.660 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:08.660 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 337 k 2023-12-07T23:44:08.660 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 1.0 M 2023-12-07T23:44:08.661 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:44:09.265 INFO:teuthology.orchestra.run.smithi184.stdout:python3-rgw-debuginfo-19.0.0-83.gc56ad3ea.el9.x 557 kB/s | 337 kB 00:00 2023-12-07T23:44:09.265 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:09.266 INFO:teuthology.orchestra.run.smithi184.stdout:Total 556 kB/s | 337 kB 00:00 2023-12-07T23:44:09.266 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:44:09.269 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:44:09.269 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:44:09.304 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:44:09.304 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:44:09.446 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:44:09.563 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:10.264 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_ 1/1 2023-12-07T23:44:10.264 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:10.264 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:10.264 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:10.264 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:10.265 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:10.466 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse-debuginfo 2023-12-07T23:44:10.822 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 16 MB/s | 51 MB 00:03 2023-12-07T23:44:10.823 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:10.824 INFO:teuthology.orchestra.run.smithi129.stdout:Total 16 MB/s | 51 MB 00:03 2023-12-07T23:44:10.824 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:44:10.828 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:44:10.828 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:44:11.049 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:23 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:44:11.161 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:44:11.161 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:44:11.199 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 508 k 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:44:11.200 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:11.201 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 508 k 2023-12-07T23:44:11.201 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.5 M 2023-12-07T23:44:11.201 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:44:11.467 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_ 1.9 MB/s | 508 kB 00:00 2023-12-07T23:44:11.468 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:11.468 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.9 MB/s | 508 kB 00:00 2023-12-07T23:44:11.468 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:44:11.472 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:44:11.472 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:44:11.508 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:44:11.508 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:44:11.666 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:44:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:12.173 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:12.449 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:12.449 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:12.449 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:12.449 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:12.450 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:12.450 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:12.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror-debuginfo 2023-12-07T23:44:13.246 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:25 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:44:13.391 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:44:13.392 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:13.392 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:44:13.392 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:13.392 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 51 M 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 51 M 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 296 M 2023-12-07T23:44:13.393 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:44:15.996 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-debuginfo-19.0.0-83.gc56ad3ea.el9.x8 20 MB/s | 51 MB 00:02 2023-12-07T23:44:15.997 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:15.997 INFO:teuthology.orchestra.run.smithi184.stdout:Total 20 MB/s | 51 MB 00:02 2023-12-07T23:44:15.997 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:44:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:44:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:44:16.332 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:44:16.332 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:44:17.016 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:44:17.112 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:44:19.255 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:44:19.530 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:44:19.531 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:19.531 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:44:19.531 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:19.531 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:19.531 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:44:19.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd-debuginfo 2023-12-07T23:44:20.308 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:09:33 ago on Thu 07 Dec 2023 11:34:47 PM UTC. 2023-12-07T23:44:20.455 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-12-07T23:44:20.455 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.2 M 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2023-12-07T23:44:20.456 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:20.457 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 1.2 M 2023-12-07T23:44:20.457 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 5.4 M 2023-12-07T23:44:20.457 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-12-07T23:44:20.775 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 3.9 MB/s | 1.2 MB 00:00 2023-12-07T23:44:20.775 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:20.776 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.9 MB/s | 1.2 MB 00:00 2023-12-07T23:44:20.776 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-12-07T23:44:20.779 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-12-07T23:44:20.779 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-12-07T23:44:20.820 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-12-07T23:44:20.820 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-12-07T23:44:21.053 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-12-07T23:44:21.157 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:21.555 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:21.846 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-12-07T23:44:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:21.846 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-12-07T23:44:22.039 DEBUG:teuthology.parallel:result is None 2023-12-07T23:44:22.147 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:44:22.243 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:44:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:44:24.629 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_6 1/1 2023-12-07T23:44:24.629 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:24.629 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:24.629 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:24.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd-debuginfo 2023-12-07T23:44:25.459 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:09:37 ago on Thu 07 Dec 2023 11:34:48 PM UTC. 2023-12-07T23:44:25.604 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-83.gc56ad3ea.el9 ceph 1.2 M 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-12-07T23:44:25.605 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-12-07T23:44:25.606 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:25.606 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 1.2 M 2023-12-07T23:44:25.606 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 5.4 M 2023-12-07T23:44:25.606 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-12-07T23:44:25.923 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-debuginfo-19.0.0-83.gc56ad3ea.el9.x86_6 3.9 MB/s | 1.2 MB 00:00 2023-12-07T23:44:25.923 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-12-07T23:44:25.924 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.9 MB/s | 1.2 MB 00:00 2023-12-07T23:44:25.924 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-12-07T23:44:25.927 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-12-07T23:44:25.927 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-12-07T23:44:25.966 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-12-07T23:44:25.967 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-12-07T23:44:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-12-07T23:44:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:26.710 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 1/1 2023-12-07T23:44:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:27.002 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-12-07T23:44:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-debuginfo-2:19.0.0-83.gc56ad3ea.el9.x86_64 2023-12-07T23:44:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:27.002 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-12-07T23:44:27.237 DEBUG:teuthology.parallel:result is None 2023-12-07T23:44:27.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-07T23:44:27.386 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-07T23:44:27.411 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-83.gc56ad3ea.el9 2023-12-07T23:44:27.412 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-83.gc56ad3ea.el9 2023-12-07T23:44:27.412 INFO:teuthology.task.install:The correct ceph version 19.0.0-83.gc56ad3ea is installed. 2023-12-07T23:44:27.412 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-07T23:44:27.553 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-07T23:44:27.574 INFO:teuthology.orchestra.run.smithi184.stdout:19.0.0-83.gc56ad3ea.el9 2023-12-07T23:44:27.575 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-83.gc56ad3ea.el9 2023-12-07T23:44:27.575 INFO:teuthology.task.install:The correct ceph version 19.0.0-83.gc56ad3ea is installed. 2023-12-07T23:44:27.575 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-07T23:44:27.575 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:27.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-07T23:44:27.607 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:44:27.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-07T23:44:27.644 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-07T23:44:27.644 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:27.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2023-12-07T23:44:27.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-07T23:44:27.748 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:44:27.748 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2023-12-07T23:44:27.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-07T23:44:27.846 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-07T23:44:27.847 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:27.847 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-07T23:44:27.876 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-07T23:44:27.945 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:44:27.945 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-07T23:44:27.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-07T23:44:28.041 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-07T23:44:28.041 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:28.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2023-12-07T23:44:28.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-07T23:44:28.138 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:44:28.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2023-12-07T23:44:28.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-07T23:44:28.235 INFO:teuthology.run_tasks:Running task ceph... 2023-12-07T23:44:28.320 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-07T23:44:28.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2023-12-07T23:44:28.323 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /var/log/ceph 2023-12-07T23:44:28.351 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-07T23:44:28.351 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-07T23:44:28.393 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-07T23:44:28.420 INFO:tasks.ceph:Creating extra log directories... 2023-12-07T23:44:28.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-07T23:44:28.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-07T23:44:28.499 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-07T23:44:28.499 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-07T23:44:28.500 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564', 'branch': 'wip-yuri3-testing-2023-12-07-0727', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7482564', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-12-07-0727'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-12-07-0727', 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9380, 'sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2023-12-07-0727', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe', 'targets': {'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk7r/sl5FVd4xb7HmAkal/vir+w88NJs5R3zxla6Lu2/yqjGtBg3JQ18Zye8EGgSE2uX+NVcIwyM+LRpgdP3FM=', 'smithi184.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA+p96ecYG4L+/ocaIhrFJzRHy5EQHa+ckbt4N23xqMZTSH7vVmpyFuMTVTnlnXutsIT1XFEzStsPoiZsgnm2k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-07_21:35:03', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-07T23:44:28.500 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-07T23:44:28.527 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-07T23:44:28.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2023-12-07T23:44:28.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2023-12-07T23:44:28.610 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:28.610 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2023-12-07T23:44:28.661 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-07T23:44:28.661 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2023-12-07T23:44:28.715 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-07T23:44:28.715 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:28.715 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-07T23:44:28.715 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:28.715 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:28.716 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-12-07 23:36:31.543416288 +0000 2023-12-07T23:44:28.716 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-12-07 23:33:56.073528872 +0000 2023-12-07T23:44:28.716 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-12-07 23:33:56.073528872 +0000 2023-12-07T23:44:28.716 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2023-12-07 23:33:56.073528872 +0000 2023-12-07T23:44:28.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-07T23:44:28.781 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-12-07T23:44:28.781 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-12-07T23:44:28.781 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000203675 s, 2.5 MB/s 2023-12-07T23:44:28.782 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-07T23:44:28.837 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2023-12-07T23:44:28.892 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-07T23:44:28.892 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:28.893 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-07T23:44:28.893 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:28.893 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:28.893 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-12-07 23:36:31.543416288 +0000 2023-12-07T23:44:28.893 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-12-07 23:33:56.563518292 +0000 2023-12-07T23:44:28.893 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-12-07 23:33:56.563518292 +0000 2023-12-07T23:44:28.893 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2023-12-07 23:33:56.563518292 +0000 2023-12-07T23:44:28.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-07T23:44:28.958 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-12-07T23:44:28.958 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-12-07T23:44:28.958 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000205292 s, 2.5 MB/s 2023-12-07T23:44:28.962 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-07T23:44:29.019 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2023-12-07T23:44:29.074 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-07T23:44:29.074 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:29.074 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-07T23:44:29.075 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:29.075 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:29.075 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-12-07 23:36:31.544416266 +0000 2023-12-07T23:44:29.075 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-12-07 23:33:57.058507604 +0000 2023-12-07T23:44:29.075 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-12-07 23:33:57.058507604 +0000 2023-12-07T23:44:29.075 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2023-12-07 23:33:57.058507604 +0000 2023-12-07T23:44:29.075 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-07T23:44:29.139 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-12-07T23:44:29.140 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-12-07T23:44:29.140 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000166547 s, 3.1 MB/s 2023-12-07T23:44:29.141 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-07T23:44:29.197 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2023-12-07T23:44:29.252 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-07T23:44:29.252 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:29.252 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-07T23:44:29.252 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:29.253 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:29.253 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-12-07 23:36:31.544416266 +0000 2023-12-07T23:44:29.253 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-12-07 23:33:57.563496700 +0000 2023-12-07T23:44:29.253 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-12-07 23:33:57.563496700 +0000 2023-12-07T23:44:29.253 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2023-12-07 23:33:57.563496700 +0000 2023-12-07T23:44:29.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-07T23:44:29.317 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-12-07T23:44:29.317 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-12-07T23:44:29.317 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000225519 s, 2.3 MB/s 2023-12-07T23:44:29.318 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-07T23:44:29.374 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:29.375 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:44:29.375 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2023-12-07T23:44:29.392 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-07T23:44:29.393 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-12-07 23:36:33.464458466 +0000 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-12-07 23:33:54.038031523 +0000 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-12-07 23:33:54.038031523 +0000 2023-12-07T23:44:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-12-07 23:33:54.038031523 +0000 2023-12-07T23:44:29.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-07T23:44:29.521 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-12-07T23:44:29.521 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-12-07T23:44:29.521 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000226096 s, 2.3 MB/s 2023-12-07T23:44:29.522 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-07T23:44:29.580 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-12-07 23:36:33.465458446 +0000 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-12-07 23:33:54.548021189 +0000 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-12-07 23:33:54.548021189 +0000 2023-12-07T23:44:29.637 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-12-07 23:33:54.548021189 +0000 2023-12-07T23:44:29.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-07T23:44:29.705 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-12-07T23:44:29.705 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-12-07T23:44:29.705 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000197881 s, 2.6 MB/s 2023-12-07T23:44:29.706 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-07T23:44:29.763 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2023-12-07T23:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-07T23:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:29.820 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-07T23:44:29.820 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:29.820 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:29.820 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-12-07 23:36:33.465458446 +0000 2023-12-07T23:44:29.820 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-12-07 23:33:55.037011280 +0000 2023-12-07T23:44:29.820 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-12-07 23:33:55.037011280 +0000 2023-12-07T23:44:29.820 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-12-07 23:33:55.037011280 +0000 2023-12-07T23:44:29.820 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-07T23:44:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-12-07T23:44:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-12-07T23:44:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000181627 s, 2.8 MB/s 2023-12-07T23:44:29.887 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-07T23:44:29.944 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-12-07 23:36:33.465458446 +0000 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-12-07 23:33:55.541001067 +0000 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-12-07 23:33:55.541001067 +0000 2023-12-07T23:44:30.001 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-12-07 23:33:55.541001067 +0000 2023-12-07T23:44:30.001 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-07T23:44:30.068 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-12-07T23:44:30.068 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-12-07T23:44:30.068 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000201418 s, 2.5 MB/s 2023-12-07T23:44:30.069 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-07T23:44:30.126 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:30.126 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-07T23:44:30.126 INFO:tasks.ceph:Generating config... 2023-12-07T23:44:30.127 INFO:tasks.ceph:[client] debug monc = 20 2023-12-07T23:44:30.127 INFO:tasks.ceph:[client] debug ms = 1 2023-12-07T23:44:30.127 INFO:tasks.ceph:[client] debug objecter = 20 2023-12-07T23:44:30.127 INFO:tasks.ceph:[client] debug rados = 20 2023-12-07T23:44:30.127 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-07T23:44:30.128 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-12-07T23:44:30.128 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-07T23:44:30.128 INFO:tasks.ceph:[global] ms type = async 2023-12-07T23:44:30.128 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-07T23:44:30.128 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-07T23:44:30.129 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-12-07T23:44:30.129 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-07T23:44:30.129 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd class default list = * 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd class load list = * 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-07T23:44:30.130 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-07T23:44:30.131 INFO:tasks.ceph:Setting up mon.a... 2023-12-07T23:44:30.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-07T23:44:30.183 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.keyring 2023-12-07T23:44:30.185 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-07T23:44:30.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T23:44:30.299 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.129'), ('mon.c', '[v2:172.21.15.129:3301,v1:172.21.15.129:6790]'), ('mon.b', '172.21.15.184')] 2023-12-07T23:44:30.299 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790],172.21.15.184', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-07T23:44:30.300 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:30.301 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-07T23:44:30.354 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.129 --addv c '[v2:172.21.15.129:3301,v1:172.21.15.129:6790]' --add b 172.21.15.184 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:30.441 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:30.441 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = pacific 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 16 (pacific) 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:election_strategy: 1 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:44:30.442 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-07T23:44:30.444 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-07T23:44:30.499 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a6eef5de-0da3-4292-875d-ef3a20551b59... 2023-12-07T23:44:30.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-07T23:44:30.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2023-12-07T23:44:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2023-12-07T23:44:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-07T23:44:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790],172.21.15.184" 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon client directed command retry = 5 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon election default strategy = 1 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject socket failures = 5000 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: ms type = async 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: osd heartbeat grace = 80 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.593 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2023-12-07T23:44:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore zero block detection = True 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 20 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 10 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: osd class default list = * 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: osd class load list = * 2023-12-07T23:44:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: osd client watch timeout = 120 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify cached snaps = True 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify missing on start = True 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd fast shutdown = False 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock override recovery settings = True 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue = debug_random 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue cut off = debug_random 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: osd shutdown pgref assert = True 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2023-12-07T23:44:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2023-12-07T23:44:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2023-12-07T23:44:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-07T23:44:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2023-12-07T23:44:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2023-12-07T23:44:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2023-12-07T23:44:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon min osdmap epochs = 50 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd crush smoke test = False 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune interval = 2 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune min = 15 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune txsize = 2 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon scrub interval = 300 2023-12-07T23:44:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on pool no app = False 2023-12-07T23:44:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: paxos service trim min = 10 2023-12-07T23:44:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:44:30.602 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2023-12-07T23:44:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2023-12-07T23:44:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2023-12-07T23:44:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2023-12-07T23:44:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-07T23:44:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-07T23:44:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 20 2023-12-07T23:44:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-12-07T23:44:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: debug objecter = 20 2023-12-07T23:44:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: debug rados = 20 2023-12-07T23:44:30.606 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2023-12-07T23:44:30.606 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2023-12-07T23:44:30.606 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2023-12-07T23:44:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2023-12-07T23:44:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2023-12-07T23:44:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2023-12-07T23:44:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2023-12-07T23:44:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2023-12-07T23:44:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = "172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790],172.21.15.184" 2023-12-07T23:44:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: mon client directed command retry = 5 2023-12-07T23:44:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: mon election default strategy = 1 2023-12-07T23:44:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: ms inject socket failures = 5000 2023-12-07T23:44:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: ms type = async 2023-12-07T23:44:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: osd heartbeat grace = 80 2023-12-07T23:44:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:44:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.612 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2023-12-07T23:44:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2023-12-07T23:44:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2023-12-07T23:44:30.613 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock profile = high_recovery_ops 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug trim objects = true 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: bdev async discard = True 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: bdev enable discard = True 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore allocator = bitmap 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore block size = 96636764160 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore fsck on mount = True 2023-12-07T23:44:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore zero block detection = True 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluefs = 1 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluestore = 1 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: debug monc = 20 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 20 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: debug rocksdb = 10 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd backfillfull_ratio = 0.85 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd full ratio = 0.9 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd nearfull ratio = 0.8 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: osd class default list = * 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: osd class load list = * 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: osd client watch timeout = 120 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify cached snaps = True 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify missing on start = True 2023-12-07T23:44:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: osd failsafe full ratio = 0.95 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: osd fast shutdown = False 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock override recovery settings = True 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: osd objectstore = bluestore 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: osd op queue = debug_random 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: osd op queue cut off = debug_random 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: osd shutdown pgref assert = True 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2023-12-07T23:44:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2023-12-07T23:44:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: auth mon ticket ttl = 660 # 11m 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: auth service ticket ttl = 240 # 4m 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: # don't complain about insecure global_id in the test suite 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-07T23:44:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: # 1m isn't quite enough 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon_down_mkfs_grace = 2m 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_filestore_osds = false 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon min osdmap epochs = 50 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd crush smoke test = False 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon osdmap full prune interval = 2 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon osdmap full prune min = 15 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon osdmap full prune txsize = 2 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon scrub interval = 300 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on pool no app = False 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: paxos service trim min = 10 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-12-07T23:44:30.620 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: debug monc = 20 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: debug objecter = 20 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: debug rados = 20 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.c] 2023-12-07T23:44:30.621 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.b] 2023-12-07T23:44:30.623 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-07T23:44:30.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-07T23:44:30.717 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-07T23:44:30.718 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:30.718 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-07T23:44:30.733 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:30.733 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-07T23:44:30.787 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2023-12-07T23:44:30.787 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:30.787 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-07T23:44:30.787 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T23:44:30.867 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:44:30.867 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:30.921 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2023-12-07T23:44:30.921 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:44:30.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-07T23:44:30.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T23:44:30.965 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:44:30.965 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:31.021 INFO:tasks.ceph:Setting up mon nodes... 2023-12-07T23:44:31.021 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-07T23:44:31.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-07T23:44:31.076 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-07T23:44:31.079 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-07T23:44:31.139 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-07T23:44:31.141 INFO:tasks.ceph:Setting up mds nodes... 2023-12-07T23:44:31.142 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-07T23:44:31.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-07T23:44:31.184 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-07T23:44:31.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-07T23:44:31.243 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-07T23:44:31.258 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-07T23:44:31.258 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-07T23:44:31.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-07T23:44:31.285 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:31.286 INFO:tasks.ceph:role: osd.0 2023-12-07T23:44:31.286 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-07T23:44:31.286 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-07T23:44:31.360 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:31.361 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:31.882 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-12-07T23:44:31.883 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-12-07T23:44:31.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-07T23:44:32.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-07T23:44:32.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-07T23:44:32.145 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:32.146 INFO:tasks.ceph:role: osd.1 2023-12-07T23:44:32.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-07T23:44:32.146 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:32.221 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:32.222 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:32.678 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-12-07T23:44:32.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-12-07T23:44:32.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-07T23:44:32.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-07T23:44:32.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-07T23:44:32.888 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:32.888 INFO:tasks.ceph:role: osd.2 2023-12-07T23:44:32.888 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-07T23:44:32.889 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-07T23:44:32.964 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:32.964 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:32.964 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:32.965 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:32.965 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:32.965 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:32.965 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:32.965 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:32.965 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:32.965 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:33.469 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-12-07T23:44:33.473 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-12-07T23:44:33.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-07T23:44:33.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-07T23:44:33.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-07T23:44:33.736 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:33.736 INFO:tasks.ceph:role: osd.3 2023-12-07T23:44:33.736 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-07T23:44:33.736 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-07T23:44:33.812 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:33.812 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:33.812 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:33.813 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:33.813 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:33.813 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:33.813 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:33.813 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:33.813 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:33.813 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:34.323 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-12-07T23:44:34.325 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-12-07T23:44:34.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-07T23:44:34.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-07T23:44:34.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:34.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:34.568+0000 7f5ad840f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-07T23:44:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:34.569+0000 7f5ad840f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-07T23:44:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:34.569+0000 7f5ad840f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T23:44:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:34.569+0000 7f5ad840f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T23:44:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:34.569+0000 7f5ad840f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T23:44:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:34.569+0000 7f5ad840f500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-07T23:44:37.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-07T23:44:37.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:37.538 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:37.535+0000 7f8499061500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-07T23:44:37.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:37.536+0000 7f8499061500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-07T23:44:37.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:37.536+0000 7f8499061500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T23:44:37.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:37.536+0000 7f8499061500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T23:44:37.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:37.536+0000 7f8499061500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T23:44:37.540 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:37.536+0000 7f8499061500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-07T23:44:40.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-07T23:44:40.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:40.505 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:40.501+0000 7fc26800d500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-07T23:44:40.505 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:40.501+0000 7fc26800d500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-07T23:44:40.505 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:40.501+0000 7fc26800d500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T23:44:40.505 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:40.501+0000 7fc26800d500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T23:44:40.505 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:40.501+0000 7fc26800d500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T23:44:40.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:40.502+0000 7fc26800d500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-07T23:44:43.351 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-07T23:44:43.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:43.470+0000 7f7b9eaf8500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-07T23:44:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:43.471+0000 7f7b9eaf8500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-07T23:44:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:43.471+0000 7f7b9eaf8500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-07T23:44:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:43.471+0000 7f7b9eaf8500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-07T23:44:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:43.471+0000 7f7b9eaf8500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-07T23:44:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:44:43.471+0000 7f7b9eaf8500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-07T23:44:46.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-07T23:44:46.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-07T23:44:46.376 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:46.377 INFO:tasks.ceph:role: osd.4 2023-12-07T23:44:46.377 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-07T23:44:46.377 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-07T23:44:46.455 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:46.455 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:46.455 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:46.456 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:46.456 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:46.456 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:46.456 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:46.456 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:46.456 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:46.456 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:46.969 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2023-12-07T23:44:46.971 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2023-12-07T23:44:46.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-07T23:44:47.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-07T23:44:47.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-07T23:44:47.225 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:47.225 INFO:tasks.ceph:role: osd.5 2023-12-07T23:44:47.225 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-07T23:44:47.226 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-07T23:44:47.301 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:47.301 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:47.301 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:47.301 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:47.301 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:47.302 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:47.302 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:47.302 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:47.302 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:47.302 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:47.744 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2023-12-07T23:44:47.746 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2023-12-07T23:44:47.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-07T23:44:47.837 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-07T23:44:47.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-07T23:44:47.935 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:47.936 INFO:tasks.ceph:role: osd.6 2023-12-07T23:44:47.936 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-07T23:44:47.936 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-07T23:44:48.010 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:48.010 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:48.010 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:48.010 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:48.010 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:48.010 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:48.010 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:48.011 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:48.011 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:48.011 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:48.503 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2023-12-07T23:44:48.505 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2023-12-07T23:44:48.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-07T23:44:48.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-07T23:44:48.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-07T23:44:48.753 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T23:44:48.753 INFO:tasks.ceph:role: osd.7 2023-12-07T23:44:48.753 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-07T23:44:48.753 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-07T23:44:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T23:44:48.827 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T23:44:48.827 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T23:44:48.828 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T23:44:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T23:44:48.828 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2023-12-07T23:44:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T23:44:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T23:44:48.828 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T23:44:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T23:44:49.343 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2023-12-07T23:44:49.345 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2023-12-07T23:44:49.345 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-07T23:44:49.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-07T23:44:49.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:49.635 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:49.632+0000 7ff4ccb82500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-07T23:44:49.635 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:49.632+0000 7ff4ccb82500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-07T23:44:49.635 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:49.632+0000 7ff4ccb82500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-07T23:44:49.636 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:49.632+0000 7ff4ccb82500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-07T23:44:49.636 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:49.632+0000 7ff4ccb82500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-07T23:44:49.636 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:49.633+0000 7ff4ccb82500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-07T23:44:52.474 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-07T23:44:52.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:52.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:52.597+0000 7f70b76d6500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-07T23:44:52.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:52.597+0000 7f70b76d6500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-07T23:44:52.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:52.597+0000 7f70b76d6500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-07T23:44:52.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:52.597+0000 7f70b76d6500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-07T23:44:52.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:52.597+0000 7f70b76d6500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-07T23:44:52.601 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:52.598+0000 7f70b76d6500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-07T23:44:55.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-07T23:44:55.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:55.567 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:55.564+0000 7fa77448c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-07T23:44:55.568 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:55.565+0000 7fa77448c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-07T23:44:55.568 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:55.565+0000 7fa77448c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-07T23:44:55.568 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:55.565+0000 7fa77448c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-07T23:44:55.568 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:55.565+0000 7fa77448c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-07T23:44:55.568 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:55.565+0000 7fa77448c500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-07T23:44:58.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-07T23:44:58.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:44:58.547 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:58.543+0000 7f5223c11500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-07T23:44:58.547 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:58.543+0000 7f5223c11500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-07T23:44:58.547 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:58.543+0000 7f5223c11500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-07T23:44:58.547 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:58.543+0000 7f5223c11500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-07T23:44:58.547 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:58.543+0000 7f5223c11500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-07T23:44:58.547 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:44:58.544+0000 7f5223c11500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-07T23:45:01.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-07T23:45:01.414 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-07T23:45:01.414 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:01.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-07T23:45:01.446 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:01.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-07T23:45:01.518 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:01.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-07T23:45:01.590 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:01.590 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-07T23:45:01.659 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:01.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-07T23:45:01.728 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:01.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-07T23:45:01.760 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:01.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-07T23:45:01.830 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:01.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-07T23:45:01.899 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:01.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-07T23:45:01.968 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:01.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-07T23:45:02.037 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:02.037 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-07T23:45:02.055 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:02.055 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-07T23:45:02.094 INFO:tasks.ceph:Adding keys to all mons... 2023-12-07T23:45:02.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-07T23:45:02.098 DEBUG:teuthology.orchestra.run.smithi184:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-07T23:45:02.137 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.y] 2023-12-07T23:45:02.137 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDfWHJlsiJ8BBAAJndjOMBkZVPv+LvDsgQY/Q== 2023-12-07T23:45:02.137 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2023-12-07T23:45:02.137 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDiWHJlWfToIRAA6OYa2gaGC8RSpmZh2PBJwQ== 2023-12-07T23:45:02.137 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2023-12-07T23:45:02.137 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDlWHJlII4AIBAAgUNVfKzkErcw4URrJM3xDw== 2023-12-07T23:45:02.137 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDoWHJl163wHRAAJJivEo9VyEJy0SXe9HdxxA== 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDrWHJllGIhHBAAkkNKDy267RdrE6/4qOPG1w== 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDfWHJl0Lo4CBAAJJG/Vv/zKuCt1oQ4AF+QnA== 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.4] 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDxWHJlbv6+JRAAqFlE4bUTUfRvk/eyqog/GQ== 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.5] 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQD0WHJll2CpIxAASQxnmn667wdJftwid+x4qA== 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.6] 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQD3WHJleb+6IRAAsjIeASZgt+NHSuEXaQBGxw== 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.7] 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQD6WHJleTJzIBAAFCw7t/0c4idGhQhLgiQIyg== 2023-12-07T23:45:02.138 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2023-12-07T23:45:02.139 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDfWHJlc1vtChAAK88wEWCEnseSweMaFcR1bg== 2023-12-07T23:45:02.139 INFO:teuthology.orchestra.run.smithi129.stdout:[client.1] 2023-12-07T23:45:02.139 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDfWHJlthtnDhAAt7jgutjJ/Cd0K+ZWvlxYwg== 2023-12-07T23:45:02.159 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.y] 2023-12-07T23:45:02.159 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDfWHJlsiJ8BBAAJndjOMBkZVPv+LvDsgQY/Q== 2023-12-07T23:45:02.159 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2023-12-07T23:45:02.159 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDiWHJlWfToIRAA6OYa2gaGC8RSpmZh2PBJwQ== 2023-12-07T23:45:02.159 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDlWHJlII4AIBAAgUNVfKzkErcw4URrJM3xDw== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDoWHJl163wHRAAJJivEo9VyEJy0SXe9HdxxA== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.3] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDrWHJllGIhHBAAkkNKDy267RdrE6/4qOPG1w== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDfWHJl0Lo4CBAAJJG/Vv/zKuCt1oQ4AF+QnA== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.4] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDxWHJlbv6+JRAAqFlE4bUTUfRvk/eyqog/GQ== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.5] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQD0WHJll2CpIxAASQxnmn667wdJftwid+x4qA== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.6] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQD3WHJleb+6IRAAsjIeASZgt+NHSuEXaQBGxw== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.7] 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQD6WHJleTJzIBAAFCw7t/0c4idGhQhLgiQIyg== 2023-12-07T23:45:02.160 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2023-12-07T23:45:02.161 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDfWHJlc1vtChAAK88wEWCEnseSweMaFcR1bg== 2023-12-07T23:45:02.161 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2023-12-07T23:45:02.161 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDfWHJlthtnDhAAt7jgutjJ/Cd0K+ZWvlxYwg== 2023-12-07T23:45:02.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T23:45:02.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T23:45:02.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T23:45:02.475 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T23:45:02.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.529 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.691 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T23:45:02.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T23:45:02.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T23:45:02.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T23:45:02.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T23:45:02.858 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-07T23:45:02.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-07T23:45:02.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-07T23:45:02.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-07T23:45:03.009 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-07T23:45:03.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-07T23:45:03.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-07T23:45:03.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-07T23:45:03.227 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-07T23:45:03.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-07T23:45:03.352 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:45:03.354 DEBUG:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-07T23:45:03.406 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-07T23:45:03.406 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-07T23:45:03.406 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-07T23:45:03.407 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-07T23:45:03.408 INFO:tasks.ceph.mon.a:Started 2023-12-07T23:45:03.409 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-07T23:45:03.409 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-07T23:45:03.409 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-07T23:45:03.411 INFO:tasks.ceph.mon.c:Started 2023-12-07T23:45:03.411 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-07T23:45:03.411 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-07T23:45:03.411 DEBUG:teuthology.orchestra.run.smithi184:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-07T23:45:03.449 INFO:tasks.ceph.mon.b:Started 2023-12-07T23:45:03.449 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-07T23:45:03.449 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-07T23:45:03.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-07T23:45:03.451 INFO:tasks.ceph.mgr.y:Started 2023-12-07T23:45:03.451 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-07T23:45:03.451 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-07T23:45:03.453 INFO:tasks.ceph.mgr.x:Started 2023-12-07T23:45:03.454 INFO:tasks.ceph:Setting crush tunables to default 2023-12-07T23:45:03.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd crush tunables default 2023-12-07T23:45:03.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.565+0000 7f09f1cc2640 10 monclient: get_monmap_and_config 2023-12-07T23:45:03.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.565+0000 7f09f1cc2640 10 monclient: build_initial_monmap 2023-12-07T23:45:03.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.565+0000 7f09f1cc2640 10 monclient: monmap: 2023-12-07T23:45:03.569 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:45:03.569 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:03.569 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:45:03.566763+0000 2023-12-07T23:45:03.569 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:45:03.566763+0000 2023-12-07T23:45:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:03.571 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:03.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.566+0000 7f09f1cc2640 1 Processor -- start 2023-12-07T23:45:03.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.566+0000 7f09f1cc2640 1 -- start start 2023-12-07T23:45:03.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.566+0000 7f09f1cc2640 10 monclient: init 2023-12-07T23:45:03.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:03.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-12-07T23:45:03.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 0x7f09ec13ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:03.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient(hunting): picked mon.noname-a con 0x7f09ec0a8970 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:03.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09ec0a9bd0 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:03.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09ec13f3b0 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec0aa950 con 0x7f09ec0a9bd0 2023-12-07T23:45:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec0a90e0 con 0x7f09ec13f3b0 2023-12-07T23:45:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0aabc0 con 0x7f09ec0a8970 2023-12-07T23:45:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09f1cc2640 10 monclient(hunting): authenticate will time out at 1303.755086s 2023-12-07T23:45:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09eb7fe640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 msgr2=0x7f09ec13ee70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2023-12-07T23:45:03.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09eaffd640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09ec0a9bd0 legacy=0x7f09ec1421a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:03.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09eb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 0x7f09ec13ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:03.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.567+0000 7f09ebfff640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09ec13f3b0 legacy=0x7f09ec14c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:03.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.768+0000 7f09eb7fe640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 msgr2=0x7f09ec13ee70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2023-12-07T23:45:03.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.768+0000 7f09eaffd640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09ec0a9bd0 legacy=0x7f09ec1421a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:03.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.768+0000 7f09eb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 0x7f09ec13ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:03.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:03.768+0000 7f09ebfff640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09ec13f3b0 legacy=0x7f09ec14c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:04.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.169+0000 7f09eb7fe640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 msgr2=0x7f09ec13ee70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2023-12-07T23:45:04.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.169+0000 7f09eaffd640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09ec0a9bd0 legacy=0x7f09ec1421a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:04.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.169+0000 7f09eb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 0x7f09ec13ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:04.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.169+0000 7f09ebfff640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09ec13f3b0 legacy=0x7f09ec14c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:04.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.970+0000 7f09eb7fe640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 msgr2=0x7f09ec13ee70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2023-12-07T23:45:04.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.970+0000 7f09eaffd640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09ec0a9bd0 legacy=0x7f09ec1421a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:04.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.970+0000 7f09eb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 0x7f09ec13ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T23:45:04.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:04.970+0000 7f09ebfff640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09ec13f3b0 legacy=0x7f09ec14c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:06.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): tick 2023-12-07T23:45:06.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): _check_auth_tickets 2023-12-07T23:45:06.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 monclient(hunting): continuing hunt 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 msgr2=0x7f09ec13ee70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09ec0a8970 0x7f09ec13ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09ec13f3b0 legacy=0x7f09ec14c780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09ec0a9bd0 legacy=0x7f09ec1421a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient: _add_conns ranks=[2,4] 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0036a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 0x7f09cc008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:06.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc0061d0 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:06.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:06.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0aabc0 con 0x7f09cc0061d0 2023-12-07T23:45:06.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:06.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc008d50 con 0x7f09cc0036a0 2023-12-07T23:45:06.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09e97fa640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:06.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:06.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:06.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09eaffd640 1 -- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 msgr2=0x7f09cc008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.184:3300/0 2023-12-07T23:45:06.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.567+0000 7f09eaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 0x7f09cc008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:06.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.768+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:06.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.768+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:06.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.768+0000 7f09eaffd640 1 -- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 msgr2=0x7f09cc008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.184:3300/0 2023-12-07T23:45:06.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:06.768+0000 7f09eaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 0x7f09cc008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:07.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.169+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:07.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.169+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:07.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.169+0000 7f09eaffd640 1 -- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 msgr2=0x7f09cc008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.184:3300/0 2023-12-07T23:45:07.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.169+0000 7f09eaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 0x7f09cc008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:07.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.970+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:07.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.970+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T23:45:07.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.970+0000 7f09eaffd640 1 -- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 msgr2=0x7f09cc008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.184:3300/0 2023-12-07T23:45:07.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:07.970+0000 7f09eaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 0x7f09cc008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): tick 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): _check_auth_tickets 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 monclient(hunting): continuing hunt 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 -- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 msgr2=0x7f09cc008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc0061d0 0x7f09cc008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-12-07T23:45:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09ebfff640 1 -- reap_dead start 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0036a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09cc005b10 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09cc008690 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec0aaea0 con 0x7f09cc008690 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec0aa950 con 0x7f09cc005b10 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0aabc0 con 0x7f09cc0036a0 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09e97fa640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09ebfff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09cc008690 legacy=0x7f09cc00cc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.567+0000 7f09eaffd640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09cc005b10 legacy=0x7f09cc009540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.768+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.768+0000 7f09ebfff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09cc008690 legacy=0x7f09cc00cc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.768+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:09.768+0000 7f09eaffd640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09cc005b10 legacy=0x7f09cc009540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.169+0000 7f09ebfff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09cc008690 legacy=0x7f09cc00cc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.169+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.169+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.169+0000 7f09eaffd640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09cc005b10 legacy=0x7f09cc009540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.970+0000 7f09eb7fe640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3301/0 2023-12-07T23:45:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.970+0000 7f09ebfff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09cc008690 legacy=0x7f09cc00cc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2023-12-07T23:45:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.970+0000 7f09eaffd640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09cc005b10 legacy=0x7f09cc009540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.184:6789/0 2023-12-07T23:45:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:10.970+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T23:45:11.119 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:00:07.564 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:45:11.404 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:00:07.877 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:45:11.404 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:00:07.877 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:45:11.432 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:00:07.877 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:45:11.718 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:00:08.191 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:45:11.719 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:00:08.192 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): tick 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): _check_auth_tickets 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 monclient(hunting): continuing hunt 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 -- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 -- >> v1:172.21.15.184:6789/0 conn(0x7f09cc005b10 legacy=0x7f09cc009540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f09cc008690 legacy=0x7f09cc00cc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient: _add_conns ranks=[3,4] 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc00f4a0 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc00ff80 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0aabc0 con 0x7f09cc00f4a0 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc008d50 con 0x7f09cc00ff80 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09e97fa640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09eb7fe640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09eb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09eaffd640 1 -- >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.184:3300/0 2023-12-07T23:45:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.567+0000 7f09eaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.768+0000 7f09eb7fe640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2023-12-07T23:45:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.768+0000 7f09eb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:12.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.841+0000 7f09eaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:12.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.860+0000 7f09eaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:34756/0 (socket says 172.21.15.129:34756) 2023-12-07T23:45:12.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.860+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 learned_addr learned my addr 172.21.15.129:0/4222164827 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.860+0000 7f09eaffd640 10 monclient(hunting): get_auth_request con 0x7f09cc00ff80 auth_method 0 2023-12-07T23:45:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.860+0000 7f09eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.860+0000 7f09eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.860+0000 7f09eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:12.898 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.895+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T23:45:12.898 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.895+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:12.898 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.895+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:12.898 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:12.895+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:13.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.296+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:13.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 10 monclient(hunting): get_auth_request con 0x7f09cc00ff80 auth_method 0 2023-12-07T23:45:13.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:13.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:13.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:13.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-07T23:45:13.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:13.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:13.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.297+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:13.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.456+0000 7f09eb7fe640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:13.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.484+0000 7f09eb7fe640 10 monclient(hunting): get_auth_request con 0x7f09cc00f4a0 auth_method 0 2023-12-07T23:45:13.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.484+0000 7f09eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:13.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.484+0000 7f09eb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:13.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.484+0000 7f09eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:13.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.486+0000 7f09eb7fe640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T23:45:13.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.486+0000 7f09eb7fe640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:13.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.486+0000 7f09eb7fe640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:13.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:13.486+0000 7f09eb7fe640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.167+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.167+0000 7f09eaffd640 10 monclient(hunting): get_auth_request con 0x7f09cc00ff80 auth_method 0 2023-12-07T23:45:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.167+0000 7f09eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.167+0000 7f09eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.167+0000 7f09eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:14.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.168+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T23:45:14.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.168+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:14.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.168+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:14.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.168+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T23:45:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.381+0000 7f09eb7fe640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:14.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.418+0000 7f09eb7fe640 10 monclient(hunting): get_auth_request con 0x7f09cc00f4a0 auth_method 0 2023-12-07T23:45:14.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.418+0000 7f09eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:14.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.418+0000 7f09eb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:14.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.418+0000 7f09eb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:14.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.422+0000 7f09eb7fe640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T23:45:14.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.422+0000 7f09eb7fe640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:14.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.422+0000 7f09eb7fe640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:14.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:14.422+0000 7f09eb7fe640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): tick 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): _check_auth_tickets 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 monclient(hunting): continuing hunt 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 msgr2=0x7f09cc00e430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.184:3300/0 conn(0x7f09cc00ff80 0x7f09cc00e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:15.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 reap_dead start 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09cc005b10 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0036a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc00f4a0 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 -- 172.21.15.129:0/4222164827 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec0a90e0 con 0x7f09cc005b10 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 -- 172.21.15.129:0/4222164827 --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0aabc0 con 0x7f09cc00f4a0 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 1 -- 172.21.15.129:0/4222164827 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc008d50 con 0x7f09cc0036a0 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.567+0000 7f09e97fa640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.568+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.568+0000 7f09eaffd640 10 monclient(hunting): get_auth_request con 0x7f09cc0036a0 auth_method 0 2023-12-07T23:45:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.568+0000 7f09eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:15.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.568+0000 7f09eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:15.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.568+0000 7f09eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:15.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.573+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:15.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.575+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-07T23:45:15.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.575+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:15.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.575+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:15.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.575+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:15.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.585+0000 7f09ebfff640 10 monclient(hunting): get_auth_request con 0x7f09cc00f4a0 auth_method 0 2023-12-07T23:45:15.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.585+0000 7f09ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:15.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.585+0000 7f09ebfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:15.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.585+0000 7f09ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:15.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.592+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-07T23:45:15.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.592+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:15.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.592+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:15.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.592+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:15.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.783+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:15.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 10 monclient(hunting): get_auth_request con 0x7f09cc0036a0 auth_method 0 2023-12-07T23:45:15.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:15.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:15.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-07T23:45:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.785+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:15.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.793+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:15.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.793+0000 7f09ebfff640 10 monclient(hunting): get_auth_request con 0x7f09cc00f4a0 auth_method 0 2023-12-07T23:45:15.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.793+0000 7f09ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:15.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.793+0000 7f09ebfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:15.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.793+0000 7f09ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:15.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.794+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-07T23:45:15.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.794+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:15.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.794+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:15.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:15.794+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T23:45:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.291+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.292+0000 7f09ebfff640 10 monclient(hunting): get_auth_request con 0x7f09cc00f4a0 auth_method 0 2023-12-07T23:45:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.292+0000 7f09ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.292+0000 7f09ebfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.292+0000 7f09ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.294+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-07T23:45:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.294+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.294+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.295+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:16.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.387+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.388+0000 7f09eaffd640 10 monclient(hunting): get_auth_request con 0x7f09cc0036a0 auth_method 0 2023-12-07T23:45:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.388+0000 7f09eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.388+0000 7f09eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.388+0000 7f09eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:16.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.499+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-07T23:45:16.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.499+0000 7f09eaffd640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:16.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.499+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:16.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.499+0000 7f09eaffd640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T23:45:16.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.665+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427212774 0 0) 0x7f09ec0a90e0 con 0x7f09cc005b10 2023-12-07T23:45:16.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.665+0000 7f09ea7fc640 10 monclient(hunting): _negotiate 2023-12-07T23:45:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.665+0000 7f09ea7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.665+0000 7f09ea7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.665+0000 7f09ea7fc640 10 monclient(hunting): my global_id is 4105 2023-12-07T23:45:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.665+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d8003520 con 0x7f09cc005b10 2023-12-07T23:45:16.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445083513 0 0) 0x7f09d8003520 con 0x7f09cc005b10 2023-12-07T23:45:16.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09cc009000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:16.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09cc009000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:16.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 msgr2=0x7f09cc00f8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:16.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 --2- 172.21.15.129:0/4222164827 >> v2:172.21.15.129:3300/0 conn(0x7f09cc00f4a0 0x7f09cc00f8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: _finish_hunting 0 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 monclient: found mon.noname-c-legacy 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.184:6789/0 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec0aaea0 con 0x7f09cc005b10 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: _finish_auth 0 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: _check_auth_tickets 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 314329121 0 0) 0x7f09dc007aa0 con 0x7f09cc005b10 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09f1cc2640 5 monclient: authenticate success, global_id 4105 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09f1cc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-07T23:45:16.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: dump: 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] but i'm connected to v1:172.21.15.184:6789/0, reconnecting 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 >> v1:172.21.15.184:6789/0 conn(0x7f09cc005b10 legacy=0x7f09cc00ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:45:16.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09d8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:16.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): picked mon.c con 0x7f09cc0036a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:16.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09d8003fa0 0x7f09d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:16.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): picked mon.a con 0x7f09d8003fa0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:16.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d8004d00 0x7f09d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): picked mon.b con 0x7f09d8004d00 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0aabc0 con 0x7f09d8003fa0 2023-12-07T23:45:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc008d50 con 0x7f09d8004d00 2023-12-07T23:45:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:16.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d800bb40 con 0x7f09cc0036a0 2023-12-07T23:45:16.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:16.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09ea7fc640 10 monclient(hunting): _finish_auth 0 2023-12-07T23:45:16.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.791+0000 7f09f1cc2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:16.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.831+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d8004d00 0x7f09d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:16.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.832+0000 7f09ebfff640 10 monclient(hunting): get_auth_request con 0x7f09d8004d00 auth_method 0 2023-12-07T23:45:16.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.832+0000 7f09ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:16.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.832+0000 7f09ebfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:16.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.832+0000 7f09ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:45:16.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.832+0000 7f09ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:16.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.832+0000 7f09ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:16.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.832+0000 7f09ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T23:45:16.837 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2023-12-07T23:45:16.837 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 msgr2=0x7f09d8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:16.837 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09cc0036a0 0x7f09d8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09d8003fa0 msgr2=0x7f09d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09d8003fa0 0x7f09d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 10 monclient: _finish_hunting 0 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 1 monclient: found mon.b 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d8003520 con 0x7f09d8004d00 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 10 monclient: _finish_auth 0 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 10 monclient: _check_auth_tickets 2023-12-07T23:45:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.834+0000 7f09ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:16.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.840+0000 7f09ebfff640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d8004d00 0x7f09d8005180 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0b5080 tx=0x7f09d80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13496a8e24bfcc7b server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4037070 con 0x7f09d8004d00 2023-12-07T23:45:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 10 monclient: dump: 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e4030430 con 0x7f09d8004d00 2023-12-07T23:45:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:16.861 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 1 -- 172.21.15.129:0/4222164827 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4030840 con 0x7f09d8004d00 2023-12-07T23:45:16.861 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:16.861 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:16.861 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09ea7fc640 10 monclient: dump: 2023-12-07T23:45:16.861 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:16.861 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:16.861 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.862 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.862 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:16.862 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:16.862 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:16.862 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:16.862 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:16.862 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:16.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09f1cc2640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:16.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.856+0000 7f09f1cc2640 4 set_mon_vals no callback set 2023-12-07T23:45:16.868 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 10 monclient: shutdown 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 1 -- 172.21.15.129:0/4222164827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d8004d00 msgr2=0x7f09d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 1 --2- 172.21.15.129:0/4222164827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09d8004d00 0x7f09d8005180 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0b5080 tx=0x7f09d80057f0 comp rx=0 tx=0).stop 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09ebfff640 1 -- 172.21.15.129:0/4222164827 reap_dead start 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 1 -- 172.21.15.129:0/4222164827 shutdown_connections 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 1 -- 172.21.15.129:0/4222164827 >> 172.21.15.129:0/4222164827 conn(0x7f09ec066dd0 msgr2=0x7f09ec0b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 1 -- 172.21.15.129:0/4222164827 shutdown_connections 2023-12-07T23:45:16.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.857+0000 7f09f1cc2640 1 -- 172.21.15.129:0/4222164827 wait complete. 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 10 monclient: build_initial_monmap 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 10 monclient: monmap: 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:16.870 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 1 librados: starting msgr at 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 1 librados: starting objecter 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 1 Processor -- start 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 1 -- start start 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 1 librados: setting wanted keys 2023-12-07T23:45:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 1 librados: calling monclient init 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.858+0000 7f09f1cc2640 10 monclient: init 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09ec1da440 0x7f09ec1da820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): picked mon.noname-a con 0x7f09ec1da440 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec0b0ad0 0x7f09ec143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): picked mon.noname-b con 0x7f09ec0b0ad0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09ec1444f0 0x7f09ec1d6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): picked mon.noname-c con 0x7f09ec1444f0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1daf50 con 0x7f09ec1da440 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec145fa0 con 0x7f09ec0b0ad0 2023-12-07T23:45:16.872 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec144970 con 0x7f09ec1444f0 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09f1cc2640 10 monclient(hunting): authenticate will time out at 1317.047391s 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.859+0000 7f09ebfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09ec1444f0 0x7f09ec1d6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.860+0000 7f09ebfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09ec1444f0 0x7f09ec1d6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:56342/0 (socket says 172.21.15.129:56342) 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.860+0000 7f09ebfff640 1 -- 172.21.15.129:0/427468994 learned_addr learned my addr 172.21.15.129:0/427468994 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.860+0000 7f09ebfff640 10 monclient(hunting): get_auth_request con 0x7f09ec1444f0 auth_method 0 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.860+0000 7f09ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.860+0000 7f09ebfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.860+0000 7f09ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.861+0000 7f09ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.861+0000 7f09ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:16.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.861+0000 7f09ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:16.874 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.867+0000 7f09eaffd640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec0b0ad0 0x7f09ec143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:16.874 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.868+0000 7f09eaffd640 10 monclient(hunting): get_auth_request con 0x7f09ec0b0ad0 auth_method 0 2023-12-07T23:45:16.874 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.868+0000 7f09eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.868+0000 7f09eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.868+0000 7f09eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.870+0000 7f09eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.870+0000 7f09eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.870+0000 7f09eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:16.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2023-12-07T23:45:16.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 1 -- 172.21.15.129:0/427468994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec0b0ad0 msgr2=0x7f09ec143fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:16.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec0b0ad0 0x7f09ec143fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:16.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 1 -- 172.21.15.129:0/427468994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09ec1da440 msgr2=0x7f09ec1da820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:16.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09ec1da440 0x7f09ec1da820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:16.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 10 monclient: _finish_hunting 0 2023-12-07T23:45:16.877 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 1 monclient: found mon.noname-c 2023-12-07T23:45:16.877 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:16.877 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:45:16.877 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.873+0000 7f09ebfff640 1 -- 172.21.15.129:0/427468994 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec144b00 con 0x7f09ec1444f0 2023-12-07T23:45:16.879 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.877+0000 7f09ebfff640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09ec1444f0 0x7f09ec1d6cd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f09e4000c00 tx=0x7f09e4005cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:16.879 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.877+0000 7f09e3fff640 10 client.?.objecter ms_handle_connect 0x7f09ec1444f0 2023-12-07T23:45:16.879 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.877+0000 7f09e3fff640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 1 -- 172.21.15.129:0/427468994 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4037040 con 0x7f09ec1444f0 2023-12-07T23:45:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: dump: 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: _finish_auth 0 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: _check_auth_tickets 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 1 -- 172.21.15.129:0/427468994 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e403c820 con 0x7f09ec1444f0 2023-12-07T23:45:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 5 monclient: authenticate success, global_id 4100 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 10 monclient: _renew_subs 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec1d7210 con 0x7f09ec1444f0 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 1 -- 172.21.15.129:0/427468994 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e403cbb0 con 0x7f09ec1444f0 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e3fff640 10 monclient: dump: 2023-12-07T23:45:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:16.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09e9ffb640 4 set_mon_vals no callback set 2023-12-07T23:45:16.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:16.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 10 monclient: _renew_subs 2023-12-07T23:45:16.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:16.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec1d76a0 con 0x7f09ec1444f0 2023-12-07T23:45:16.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.938+0000 7f09f1cc2640 1 librados: init done 2023-12-07T23:45:16.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.939+0000 7f09f1cc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:16.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.939+0000 7f09f1cc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:16.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.939+0000 7f09f1cc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:16.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:16.939+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec144c90 con 0x7f09ec1444f0 2023-12-07T23:45:17.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 1 -- 172.21.15.129:0/427468994 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f09e4046740 con 0x7f09ec1444f0 2023-12-07T23:45:17.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 1 -- 172.21.15.129:0/427468994 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f09e404f420 con 0x7f09ec1444f0 2023-12-07T23:45:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 10 client.4100.objecter ms_dispatch 0x7f09ec000b90 osd_map(1..1 src has 1..1) v4 2023-12-07T23:45:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2023-12-07T23:45:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 3 client.4100.objecter handle_osd_map decoding full epoch 1 2023-12-07T23:45:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 20 client.4100.objecter dump_active .. 0 homeless 2023-12-07T23:45:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 1 -- 172.21.15.129:0/427468994 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f09e404b060 con 0x7f09ec1444f0 2023-12-07T23:45:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.134+0000 7f09e3fff640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:17.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.221+0000 7f09f1cc2640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-07T23:45:17.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.221+0000 7f09f1cc2640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-07T23:45:17.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.221+0000 7f09f1cc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:17.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.221+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f09ec04e840 con 0x7f09ec1444f0 2023-12-07T23:45:17.439 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:17.436+0000 7f57d6813040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-07T23:45:17.491 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:17.488+0000 7f57d6813040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-07T23:45:17.540 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:17.538+0000 7f6055e4b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-07T23:45:17.594 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:17.593+0000 7f6055e4b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-07T23:45:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.609+0000 7f09e3fff640 1 -- 172.21.15.129:0/427468994 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f09e4005840 con 0x7f09ec1444f0 2023-12-07T23:45:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.609+0000 7f09e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-07T23:45:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.609+0000 7f09e3fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-12-07T23:45:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2023-12-07T23:45:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.609+0000 7f09f1cc2640 10 librados: watch_flush enter 2023-12-07T23:45:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 10 librados: watch_flush exit 2023-12-07T23:45:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 20 client.4100.objecter shutdown clearing up homeless session... 2023-12-07T23:45:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 10 client.4100.objecter successfully canceled tick 2023-12-07T23:45:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 10 monclient: shutdown 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09ec1444f0 msgr2=0x7f09ec1d6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09ec1444f0 0x7f09ec1d6cd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f09e4000c00 tx=0x7f09e4005cc0 comp rx=0 tx=0).stop 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 shutdown_connections 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f09ec1444f0 0x7f09ec1d6cd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec0b0ad0 0x7f09ec143fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 --2- 172.21.15.129:0/427468994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09ec1da440 0x7f09ec1da820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 >> 172.21.15.129:0/427468994 conn(0x7f09ec066dd0 msgr2=0x7f09ec0b6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 shutdown_connections 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 -- 172.21.15.129:0/427468994 wait complete. 2023-12-07T23:45:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:17.610+0000 7f09f1cc2640 1 librados: shutdown 2023-12-07T23:45:17.622 INFO:tasks.ceph:check_enable_crimson: False 2023-12-07T23:45:17.623 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-07T23:45:17.623 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:17.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-07T23:45:17.655 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:17.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-07T23:45:17.723 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:17.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-07T23:45:17.791 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:45:17.791 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-07T23:45:17.857 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:17.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-07T23:45:17.885 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:17.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-07T23:45:17.952 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:17.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-07T23:45:18.019 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-12-07T23:45:18.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-07T23:45:18.045 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.042+0000 7f57d6813040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-07T23:45:18.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new cd51c7b8-344d-44a8-9d6d-79c958dbaa17 0 2023-12-07T23:45:18.099 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.097+0000 7f57d6813040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-07T23:45:18.147 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.144+0000 7f57d6813040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-07T23:45:18.170 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.168+0000 7f6055e4b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-07T23:45:18.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.197+0000 7f2de897d640 10 monclient: get_monmap_and_config 2023-12-07T23:45:18.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.197+0000 7f2de897d640 10 monclient: build_initial_monmap 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.197+0000 7f2de897d640 10 monclient: monmap: 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:18.197983+0000 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:18.197983+0000 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:18.200 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.197+0000 7f2de897d640 1 Processor -- start 2023-12-07T23:45:18.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.197+0000 7f2de897d640 1 -- start start 2023-12-07T23:45:18.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.197+0000 7f2de897d640 10 monclient: init 2023-12-07T23:45:18.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de40a8970 0x7f2de413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2de40a9bd0 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2de413f3b0 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de40aa950 con 0x7f2de413f3b0 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de413f840 con 0x7f2de40a9bd0 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40a90e0 con 0x7f2de40a8970 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.198+0000 7f2de897d640 10 monclient(hunting): authenticate will time out at 1323.511819s 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.199+0000 7f2ddbfff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f2de40a9bd0 0x7f2de41421a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:60948/0 (socket says 172.21.15.184:60948) 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.199+0000 7f2ddbfff640 1 -- 172.21.15.184:0/1535632453 learned_addr learned my addr 172.21.15.184:0/1535632453 (peer_addr_for_me v1:172.21.15.184:0/0) 2023-12-07T23:45:18.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.199+0000 7f2de259c640 1 --2- 172.21.15.184:0/1535632453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de40a8970 0x7f2de413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.199+0000 7f2de259c640 10 monclient(hunting): get_auth_request con 0x7f2de40a8970 auth_method 0 2023-12-07T23:45:18.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.199+0000 7f2de259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.199+0000 7f2de259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.199+0000 7f2de259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.200+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.200+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.200+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.205 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.201+0000 7f57d6813040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-07T23:45:18.205 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de1d9b640 1 -- 172.21.15.184:0/1535632453 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297572805 0 0) 0x7f2de413f840 con 0x7f2de40a9bd0 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de1d9b640 10 monclient(hunting): _negotiate 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de1d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de1d9b640 10 monclient(hunting): my global_id is 4117 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de1d9b640 1 -- 172.21.15.184:0/1535632453 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd0003520 con 0x7f2de40a9bd0 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de259c640 10 monclient(hunting): handle_auth_done global_id 4103 payload 274 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.202+0000 7f2de259c640 1 -- 172.21.15.184:0/1535632453 >> v1:172.21.15.184:6789/0 conn(0x7f2de40a9bd0 legacy=0x7f2de41421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.203+0000 7f2de259c640 1 -- 172.21.15.184:0/1535632453 >> v1:172.21.15.129:6789/0 conn(0x7f2de413f3b0 legacy=0x7f2de414c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.203+0000 7f2de259c640 10 monclient: _finish_hunting 0 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.203+0000 7f2de259c640 1 monclient: found mon.noname-b 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.203+0000 7f2de259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.203+0000 7f2de259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:45:18.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.203+0000 7f2de259c640 1 -- 172.21.15.184:0/1535632453 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de40aabc0 con 0x7f2de40a8970 2023-12-07T23:45:18.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.205+0000 7f2de259c640 1 --2- 172.21.15.184:0/1535632453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de40a8970 0x7f2de413ee70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40097d0 tx=0x7f2dd402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2fa22fb3c3793fa server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 1 -- 172.21.15.184:0/1535632453 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4031770 con 0x7f2de40a8970 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: dump: 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.214 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: _finish_auth 0 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: _check_auth_tickets 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 1 -- 172.21.15.184:0/1535632453 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd40318d0 con 0x7f2de40a8970 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 1 -- 172.21.15.184:0/1535632453 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4031c60 con 0x7f2de40a8970 2023-12-07T23:45:18.215 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de1d9b640 10 monclient: dump: 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.216 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de897d640 5 monclient: authenticate success, global_id 4103 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de897d640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.210+0000 7f2de897d640 4 set_mon_vals no callback set 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 10 monclient: shutdown 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 1 -- 172.21.15.184:0/1535632453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de40a8970 msgr2=0x7f2de413ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 1 --2- 172.21.15.184:0/1535632453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de40a8970 0x7f2de413ee70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40097d0 tx=0x7f2dd402b810 comp rx=0 tx=0).stop 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 1 -- 172.21.15.184:0/1535632453 shutdown_connections 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 1 --2- 172.21.15.184:0/1535632453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de40a8970 0x7f2de413ee70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 1 -- 172.21.15.184:0/1535632453 >> 172.21.15.184:0/1535632453 conn(0x7f2de4066dd0 msgr2=0x7f2de40b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:18.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 1 -- 172.21.15.184:0/1535632453 shutdown_connections 2023-12-07T23:45:18.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.211+0000 7f2de897d640 1 -- 172.21.15.184:0/1535632453 wait complete. 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 10 monclient: build_initial_monmap 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 10 monclient: monmap: 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:18.222 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 1 librados: starting msgr at 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 1 librados: starting objecter 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 1 Processor -- start 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 1 -- start start 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 1 librados: setting wanted keys 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 1 librados: calling monclient init 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.212+0000 7f2de897d640 10 monclient: init 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2de40a8970 0x7f2de4144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): picked mon.noname-a con 0x7f2de40a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:18.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2de40a9bd0 0x7f2de41da440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40a9bd0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de413f3b0 0x7f2de41da980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): picked mon.noname-c con 0x7f2de413f3b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4145980 con 0x7f2de40a8970 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4145ea0 con 0x7f2de40a9bd0 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4146060 con 0x7f2de413f3b0 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:18.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de897d640 10 monclient(hunting): authenticate will time out at 1323.526409s 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2de40a9bd0 0x7f2de41da440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.213+0000 7f2de2d9d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de413f3b0 0x7f2de41da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2de40a9bd0 0x7f2de41da440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58826/0 (socket says 172.21.15.184:58826) 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 1 -- 172.21.15.184:0/4103848936 learned_addr learned my addr 172.21.15.184:0/4103848936 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 10 monclient(hunting): get_auth_request con 0x7f2de40a9bd0 auth_method 0 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2de2d9d640 10 monclient(hunting): get_auth_request con 0x7f2de413f3b0 auth_method 0 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2de2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2de2d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2de2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.214+0000 7f2ddbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.215+0000 7f2de2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.215+0000 7f2de2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.215+0000 7f2de2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2de259c640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2de40a8970 0x7f2de4144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 1 -- 172.21.15.184:0/4103848936 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de413f3b0 msgr2=0x7f2de41da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de413f3b0 0x7f2de41da980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 1 -- 172.21.15.184:0/4103848936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2de40a8970 msgr2=0x7f2de4144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2de40a8970 0x7f2de4144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 10 monclient: _finish_hunting 0 2023-12-07T23:45:18.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 1 monclient: found mon.noname-b 2023-12-07T23:45:18.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:18.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.217+0000 7f2ddbfff640 1 -- 172.21.15.184:0/4103848936 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4009400 con 0x7f2de40a9bd0 2023-12-07T23:45:18.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.219+0000 7f2ddbfff640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2de40a9bd0 0x7f2de41da440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00c4d0 tx=0x7f2dcc00c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:18.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.219+0000 7f2ddaffd640 10 client.?.objecter ms_handle_connect 0x7f2de40a9bd0 2023-12-07T23:45:18.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.219+0000 7f2ddaffd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:18.228 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 1 -- 172.21.15.184:0/4103848936 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc005ce0 con 0x7f2de40a9bd0 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 10 monclient: dump: 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.229 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 10 monclient: _finish_auth 0 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 10 monclient: _check_auth_tickets 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 1 -- 172.21.15.184:0/4103848936 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc005e40 con 0x7f2de40a9bd0 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2de897d640 5 monclient: authenticate success, global_id 4120 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2de897d640 10 monclient: _renew_subs 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2de897d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de4146260 con 0x7f2de40a9bd0 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.222+0000 7f2ddaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2ddaffd640 1 -- 172.21.15.184:0/4103848936 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc007270 con 0x7f2de40a9bd0 2023-12-07T23:45:18.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2ddaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2ddaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2ddaffd640 10 monclient: dump: 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2de159a640 4 set_mon_vals no callback set 2023-12-07T23:45:18.231 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2de897d640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2de897d640 10 monclient: _renew_subs 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2de897d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de41453d0 con 0x7f2de40a9bd0 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.223+0000 7f2de897d640 1 librados: init done 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.224+0000 7f2de897d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.224+0000 7f2de897d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.224+0000 7f2de897d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.224+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de413e4a0 con 0x7f2de40a9bd0 2023-12-07T23:45:18.233 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.230+0000 7f6055e4b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-07T23:45:18.233 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.226+0000 7f2ddaffd640 1 -- 172.21.15.184:0/4103848936 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2dcc01c440 con 0x7f2de40a9bd0 2023-12-07T23:45:18.233 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.226+0000 7f2ddaffd640 1 -- 172.21.15.184:0/4103848936 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2dcc012740 con 0x7f2de40a9bd0 2023-12-07T23:45:18.233 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.226+0000 7f2ddaffd640 10 client.4120.objecter ms_dispatch 0x7f2de4000b90 osd_map(2..2 src has 1..2) v4 2023-12-07T23:45:18.234 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.226+0000 7f2ddaffd640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2023-12-07T23:45:18.234 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.226+0000 7f2ddaffd640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2023-12-07T23:45:18.234 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.226+0000 7f2ddaffd640 20 client.4120.objecter dump_active .. 0 homeless 2023-12-07T23:45:18.248 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.245+0000 7f57d6813040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-07T23:45:18.276 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.274+0000 7f6055e4b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-07T23:45:18.328 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.327+0000 7f6055e4b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-07T23:45:18.349 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.346+0000 7f57d6813040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-07T23:45:18.369 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.368+0000 7f6055e4b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-07T23:45:18.396 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.393+0000 7f2ddaffd640 1 -- 172.21.15.184:0/4103848936 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2dcc021060 con 0x7f2de40a9bd0 2023-12-07T23:45:18.396 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.393+0000 7f2ddaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:18.396 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.393+0000 7f2ddaffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:18.419 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.416+0000 7f57d6813040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-07T23:45:18.466 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.463+0000 7f57d6813040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-07T23:45:18.468 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.466+0000 7f2de897d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cd51c7b8-344d-44a8-9d6d-79c958dbaa17", "id": 0}] 2023-12-07T23:45:18.468 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.466+0000 7f2de897d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cd51c7b8-344d-44a8-9d6d-79c958dbaa17", "id": 0}] 2023-12-07T23:45:18.468 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.466+0000 7f2de897d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.468 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.466+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cd51c7b8-344d-44a8-9d6d-79c958dbaa17", "id": 0} v 0) v1 -- 0x7f2de404e840 con 0x7f2de40a9bd0 2023-12-07T23:45:18.469 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.467+0000 7f6055e4b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-07T23:45:18.518 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.516+0000 7f57d6813040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-07T23:45:18.539 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.537+0000 7f6055e4b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-07T23:45:18.565 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.562+0000 7f57d6813040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-07T23:45:18.590 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.588+0000 7f6055e4b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-07T23:45:18.646 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.644+0000 7f6055e4b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-07T23:45:18.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.652+0000 7f2ddaffd640 1 -- 172.21.15.184:0/4103848936 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cd51c7b8-344d-44a8-9d6d-79c958dbaa17", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2dcc017070 con 0x7f2de40a9bd0 2023-12-07T23:45:18.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.652+0000 7f2ddaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cd51c7b8-344d-44a8-9d6d-79c958dbaa17", "id": 0}] 2023-12-07T23:45:18.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.652+0000 7f2ddaffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:18.656 INFO:teuthology.orchestra.run.smithi184.stdout:0 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 10 librados: watch_flush enter 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 10 librados: watch_flush exit 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 20 client.4120.objecter shutdown clearing up homeless session... 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 10 client.4120.objecter successfully canceled tick 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 10 monclient: shutdown 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2de40a9bd0 msgr2=0x7f2de41da440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2de40a9bd0 0x7f2de41da440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00c4d0 tx=0x7f2dcc00c990 comp rx=0 tx=0).stop 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 shutdown_connections 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2de413f3b0 0x7f2de41da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2de40a9bd0 0x7f2de41da440 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 --2- 172.21.15.184:0/4103848936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2de40a8970 0x7f2de4144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 >> 172.21.15.184:0/4103848936 conn(0x7f2de4066dd0 msgr2=0x7f2de40b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 shutdown_connections 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 -- 172.21.15.184:0/4103848936 wait complete. 2023-12-07T23:45:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.653+0000 7f2de897d640 1 librados: shutdown 2023-12-07T23:45:18.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 81a06079-4753-4cb3-8d07-858bf1ee6e4e 1 2023-12-07T23:45:18.702 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:18.700+0000 7f6055e4b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-07T23:45:18.768 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.765+0000 7f2db5c39640 10 monclient: get_monmap_and_config 2023-12-07T23:45:18.768 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.765+0000 7f2db5c39640 10 monclient: build_initial_monmap 2023-12-07T23:45:18.768 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.765+0000 7f2db5c39640 10 monclient: monmap: 2023-12-07T23:45:18.768 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:18.768 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:18.766650+0000 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:18.766650+0000 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:18.769 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.771 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.766+0000 7f2db5c39640 1 Processor -- start 2023-12-07T23:45:18.771 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.766+0000 7f2db5c39640 1 -- start start 2023-12-07T23:45:18.771 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.766+0000 7f2db5c39640 10 monclient: init 2023-12-07T23:45:18.771 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:18.771 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db00a8970 0x7f2db013ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient(hunting): picked mon.noname-b con 0x7f2db00a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2db00a9bd0 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2db013f3b0 0x7f2db01428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient(hunting): picked mon.noname-a con 0x7f2db013f3b0 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db00aa950 con 0x7f2db00a9bd0 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db0142df0 con 0x7f2db013f3b0 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db013f840 con 0x7f2db00a8970 2023-12-07T23:45:18.772 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.767+0000 7f2db5c39640 10 monclient(hunting): authenticate will time out at 1324.080578s 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daffff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2db013f3b0 0x7f2db01428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daf7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db00a8970 0x7f2db013ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daffff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2db013f3b0 0x7f2db01428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.184:37198/0 (socket says 172.21.15.184:37198) 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daffff640 1 -- 172.21.15.184:0/79574676 learned_addr learned my addr 172.21.15.184:0/79574676 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daffff640 10 monclient(hunting): get_auth_request con 0x7f2db013f3b0 auth_method 0 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daf7fe640 10 monclient(hunting): get_auth_request con 0x7f2db00a8970 auth_method 0 2023-12-07T23:45:18.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.768+0000 7f2daf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.769+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.769+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.769+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.769+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.769+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.769+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.775 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 10 monclient(hunting): handle_auth_done global_id 4104 payload 274 2023-12-07T23:45:18.775 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 1 -- 172.21.15.184:0/79574676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db00a8970 msgr2=0x7f2db013ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:18.775 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 1 --2- 172.21.15.184:0/79574676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db00a8970 0x7f2db013ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.775 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 1 -- 172.21.15.184:0/79574676 >> v1:172.21.15.129:6789/0 conn(0x7f2db00a9bd0 legacy=0x7f2db01421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:18.775 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 10 monclient: _finish_hunting 0 2023-12-07T23:45:18.776 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 1 monclient: found mon.noname-a 2023-12-07T23:45:18.776 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:18.776 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:45:18.776 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.770+0000 7f2daffff640 1 -- 172.21.15.184:0/79574676 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db00a90e0 con 0x7f2db013f3b0 2023-12-07T23:45:18.779 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.777+0000 7f2daffff640 1 --2- 172.21.15.184:0/79574676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db013f3b0 0x7f2db01428b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009e50 tx=0x7f2da802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68796418518b209 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:18.816 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 1 -- 172.21.15.184:0/79574676 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da802c950 con 0x7f2db013f3b0 2023-12-07T23:45:18.816 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.816 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:45:18.816 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: dump: 2023-12-07T23:45:18.816 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.816 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: _finish_auth 0 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: _check_auth_tickets 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 1 -- 172.21.15.184:0/79574676 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da802cab0 con 0x7f2db013f3b0 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2db5c39640 5 monclient: authenticate success, global_id 4104 2023-12-07T23:45:18.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2db5c39640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2db5c39640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 1 -- 172.21.15.184:0/79574676 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da802ce40 con 0x7f2db013f3b0 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2db5c39640 4 set_mon_vals no callback set 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.813+0000 7f2daeffd640 10 monclient: dump: 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.818 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.819 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.819 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 10 monclient: shutdown 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 -- 172.21.15.184:0/79574676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db013f3b0 msgr2=0x7f2db01428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 --2- 172.21.15.184:0/79574676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db013f3b0 0x7f2db01428b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009e50 tx=0x7f2da802bdb0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 -- 172.21.15.184:0/79574676 shutdown_connections 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 --2- 172.21.15.184:0/79574676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db013f3b0 0x7f2db01428b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 --2- 172.21.15.184:0/79574676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db00a8970 0x7f2db013ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 -- 172.21.15.184:0/79574676 >> 172.21.15.184:0/79574676 conn(0x7f2db0066dd0 msgr2=0x7f2db00b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 -- 172.21.15.184:0/79574676 shutdown_connections 2023-12-07T23:45:18.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.814+0000 7f2db5c39640 1 -- 172.21.15.184:0/79574676 wait complete. 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 10 monclient: build_initial_monmap 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 10 monclient: monmap: 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.823 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 1 librados: starting msgr at 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 1 librados: starting objecter 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 1 Processor -- start 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 1 -- start start 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 1 librados: setting wanted keys 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 1 librados: calling monclient init 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.815+0000 7f2db5c39640 10 monclient: init 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00a8970 0x7f2db01da980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): picked mon.noname-b con 0x7f2db00a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db00a9bd0 0x7f2db0137aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): picked mon.noname-a con 0x7f2db00a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db013f3b0 0x7f2db00b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:18.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): picked mon.noname-c con 0x7f2db013f3b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00abd70 con 0x7f2db00a9bd0 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db013a770 con 0x7f2db00a8970 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00abf70 con 0x7f2db013f3b0 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.816+0000 7f2db5c39640 10 monclient(hunting): authenticate will time out at 1324.129801s 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00a8970 0x7f2db01da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2da6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db00a9bd0 0x7f2db0137aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daffff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db013f3b0 0x7f2db00b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00a8970 0x7f2db01da980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58838/0 (socket says 172.21.15.184:58838) 2023-12-07T23:45:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daf7fe640 1 -- 172.21.15.184:0/2451335744 learned_addr learned my addr 172.21.15.184:0/2451335744 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daf7fe640 10 monclient(hunting): get_auth_request con 0x7f2db00a8970 auth_method 0 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2da6ffd640 10 monclient(hunting): get_auth_request con 0x7f2db00a9bd0 auth_method 0 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2da6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2da6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daffff640 10 monclient(hunting): get_auth_request con 0x7f2db013f3b0 auth_method 0 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.817+0000 7f2daffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.827 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2da6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.827 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2da6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.827 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2da6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.827 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:18.827 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:18.827 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.818+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 1 -- 172.21.15.184:0/2451335744 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db013f3b0 msgr2=0x7f2db00b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 1 --2- 172.21.15.184:0/2451335744 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db013f3b0 0x7f2db00b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 1 -- 172.21.15.184:0/2451335744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db00a9bd0 msgr2=0x7f2db0137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 1 --2- 172.21.15.184:0/2451335744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db00a9bd0 0x7f2db0137aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 10 monclient: _finish_hunting 0 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 1 monclient: found mon.noname-b 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.820+0000 7f2daf7fe640 1 -- 172.21.15.184:0/2451335744 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8009b30 con 0x7f2db00a8970 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.822+0000 7f2daf7fe640 1 --2- 172.21.15.184:0/2451335744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00a8970 0x7f2db01da980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9b0 tx=0x7f2da000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:18.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.822+0000 7f2dacff9640 10 client.?.objecter ms_handle_connect 0x7f2db00a8970 2023-12-07T23:45:18.829 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.822+0000 7f2dacff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 1 -- 172.21.15.184:0/2451335744 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0009670 con 0x7f2db00a8970 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: dump: 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.830 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: _finish_auth 0 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: _check_auth_tickets 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 1 -- 172.21.15.184:0/2451335744 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da000c9a0 con 0x7f2db00a8970 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 1 -- 172.21.15.184:0/2451335744 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da000cd50 con 0x7f2db00a8970 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dacff9640 10 monclient: dump: 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:18.831 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 5 monclient: authenticate success, global_id 4123 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 10 monclient: _renew_subs 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2dae7fc640 4 set_mon_vals no callback set 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db00ac250 con 0x7f2db00a8970 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 10 monclient: _renew_subs 2023-12-07T23:45:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db013a4e0 con 0x7f2db00a8970 2023-12-07T23:45:18.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.826+0000 7f2db5c39640 1 librados: init done 2023-12-07T23:45:18.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.827+0000 7f2db5c39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:18.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.827+0000 7f2db5c39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:18.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.827+0000 7f2db5c39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.827+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db004e840 con 0x7f2db00a8970 2023-12-07T23:45:18.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.830+0000 7f2dacff9640 1 -- 172.21.15.184:0/2451335744 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2da0005c90 con 0x7f2db00a8970 2023-12-07T23:45:18.834 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.830+0000 7f2dacff9640 1 -- 172.21.15.184:0/2451335744 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f2da0013470 con 0x7f2db00a8970 2023-12-07T23:45:18.834 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.830+0000 7f2dacff9640 10 client.4123.objecter ms_dispatch 0x7f2db0000b90 osd_map(3..3 src has 1..3) v4 2023-12-07T23:45:18.834 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.830+0000 7f2dacff9640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2023-12-07T23:45:18.834 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.830+0000 7f2dacff9640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2023-12-07T23:45:18.834 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.830+0000 7f2dacff9640 20 client.4123.objecter dump_active .. 0 homeless 2023-12-07T23:45:18.926 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.923+0000 7f2dacff9640 1 -- 172.21.15.184:0/2451335744 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2da0021060 con 0x7f2db00a8970 2023-12-07T23:45:18.926 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.923+0000 7f2dacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:18.926 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.923+0000 7f2dacff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:18.956 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:18.953+0000 7f57d6813040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-07T23:45:18.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.996+0000 7f2db5c39640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "81a06079-4753-4cb3-8d07-858bf1ee6e4e", "id": 1}] 2023-12-07T23:45:18.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.996+0000 7f2db5c39640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "81a06079-4753-4cb3-8d07-858bf1ee6e4e", "id": 1}] 2023-12-07T23:45:18.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.996+0000 7f2db5c39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:18.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:18.996+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "81a06079-4753-4cb3-8d07-858bf1ee6e4e", "id": 1} v 0) v1 -- 0x7f2db013c440 con 0x7f2db00a8970 2023-12-07T23:45:19.079 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.077+0000 7f2dacff9640 1 -- 172.21.15.184:0/2451335744 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "81a06079-4753-4cb3-8d07-858bf1ee6e4e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2da0017070 con 0x7f2db00a8970 2023-12-07T23:45:19.080 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.077+0000 7f2dacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "81a06079-4753-4cb3-8d07-858bf1ee6e4e", "id": 1}] 2023-12-07T23:45:19.080 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.077+0000 7f2dacff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:19.080 INFO:teuthology.orchestra.run.smithi184.stdout:1 2023-12-07T23:45:19.081 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.077+0000 7f2db5c39640 10 librados: watch_flush enter 2023-12-07T23:45:19.081 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 10 librados: watch_flush exit 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 20 client.4123.objecter shutdown clearing up homeless session... 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 10 client.4123.objecter successfully canceled tick 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 10 monclient: shutdown 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00a8970 msgr2=0x7f2db01da980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 --2- 172.21.15.184:0/2451335744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00a8970 0x7f2db01da980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9b0 tx=0x7f2da000ee70 comp rx=0 tx=0).stop 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 shutdown_connections 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 --2- 172.21.15.184:0/2451335744 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2db013f3b0 0x7f2db00b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 --2- 172.21.15.184:0/2451335744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2db00a9bd0 0x7f2db0137aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 --2- 172.21.15.184:0/2451335744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00a8970 0x7f2db01da980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 >> 172.21.15.184:0/2451335744 conn(0x7f2db0066dd0 msgr2=0x7f2db00b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 shutdown_connections 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 -- 172.21.15.184:0/2451335744 wait complete. 2023-12-07T23:45:19.082 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.078+0000 7f2db5c39640 1 librados: shutdown 2023-12-07T23:45:19.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new b8306026-161e-40a0-bf4b-856ecf0b4d17 2 2023-12-07T23:45:19.093 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.091+0000 7f57d6813040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-07T23:45:19.096 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.094+0000 7f6055e4b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-07T23:45:19.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.188+0000 7f172b71d640 10 monclient: get_monmap_and_config 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.188+0000 7f172b71d640 10 monclient: build_initial_monmap 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.188+0000 7f172b71d640 10 monclient: monmap: 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:19.189034+0000 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:19.189034+0000 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:19.194 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.188+0000 7f172b71d640 1 Processor -- start 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.188+0000 7f172b71d640 1 -- start start 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.188+0000 7f172b71d640 10 monclient: init 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17240a8970 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f17240a9bd0 0x7f17241421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient(hunting): picked mon.noname-c con 0x7f17240a9bd0 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 0x7f17241426e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient(hunting): picked mon.noname-b con 0x7f172413f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17240aa950 con 0x7f17240a8970 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17240a90e0 con 0x7f17240a9bd0 2023-12-07T23:45:19.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17240aabc0 con 0x7f172413f580 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.189+0000 7f172b71d640 10 monclient(hunting): authenticate will time out at 1324.502850s 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f17240a9bd0 0x7f17241421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 0x7f17241426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f17240a9bd0 0x7f17241421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58844/0 (socket says 172.21.15.184:58844) 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 1 -- 172.21.15.184:0/945043691 learned_addr learned my addr 172.21.15.184:0/945043691 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 10 monclient(hunting): get_auth_request con 0x7f17240a9bd0 auth_method 0 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 10 monclient(hunting): get_auth_request con 0x7f172413f580 auth_method 0 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.196 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 1 -- 172.21.15.184:0/945043691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 msgr2=0x7f17241426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-07T23:45:19.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 1 -- 172.21.15.184:0/945043691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 msgr2=0x7f17241426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T23:45:19.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 1 --2- 172.21.15.184:0/945043691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 0x7f17241426e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T23:45:19.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1729cba640 1 --2- 172.21.15.184:0/945043691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 0x7f17241426e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T23:45:19.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.190+0000 7f1728cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2023-12-07T23:45:19.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 1 -- 172.21.15.184:0/945043691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 msgr2=0x7f17241426e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 1 --2- 172.21.15.184:0/945043691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 0x7f17241426e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 1 -- 172.21.15.184:0/945043691 >> v1:172.21.15.184:6789/0 conn(0x7f17240a8970 legacy=0x7f172413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 10 monclient: _finish_hunting 0 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 1 monclient: found mon.noname-c 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.192+0000 7f1728cb8640 1 -- 172.21.15.184:0/945043691 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17240aaea0 con 0x7f17240a9bd0 2023-12-07T23:45:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.194+0000 7f1728cb8640 1 --2- 172.21.15.184:0/945043691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17240a9bd0 0x7f17241421a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1720009870 tx=0x7f172002b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45e1319eef7e033d server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:19.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 1 -- 172.21.15.184:0/945043691 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720031740 con 0x7f17240a9bd0 2023-12-07T23:45:19.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:19.204 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: dump: 2023-12-07T23:45:19.204 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: _finish_auth 0 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: _check_auth_tickets 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 1 -- 172.21.15.184:0/945043691 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17200318a0 con 0x7f17240a9bd0 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f172b71d640 5 monclient: authenticate success, global_id 4126 2023-12-07T23:45:19.205 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f172b71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f172b71d640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 1 -- 172.21.15.184:0/945043691 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720031c30 con 0x7f17240a9bd0 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f172b71d640 4 set_mon_vals no callback set 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.200+0000 7f171bfff640 10 monclient: dump: 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.206 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 10 monclient: shutdown 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 -- 172.21.15.184:0/945043691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17240a9bd0 msgr2=0x7f17241421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.207 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 --2- 172.21.15.184:0/945043691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17240a9bd0 0x7f17241421a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1720009870 tx=0x7f172002b7e0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 -- 172.21.15.184:0/945043691 shutdown_connections 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 --2- 172.21.15.184:0/945043691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f172413f580 0x7f17241426e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 --2- 172.21.15.184:0/945043691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17240a9bd0 0x7f17241421a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 -- 172.21.15.184:0/945043691 >> 172.21.15.184:0/945043691 conn(0x7f1724066dd0 msgr2=0x7f17240b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 -- 172.21.15.184:0/945043691 shutdown_connections 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.201+0000 7f172b71d640 1 -- 172.21.15.184:0/945043691 wait complete. 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 10 monclient: build_initial_monmap 2023-12-07T23:45:19.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 10 monclient: monmap: 2023-12-07T23:45:19.209 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:19.209 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:19.209 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:19.209 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:19.209 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:19.209 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.209 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:19.210 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:19.210 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:19.210 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 1 librados: starting msgr at 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 1 librados: starting objecter 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 1 Processor -- start 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 1 -- start start 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 1 librados: setting wanted keys 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 1 librados: calling monclient init 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.202+0000 7f172b71d640 10 monclient: init 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17240a8970 0x7f172413b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): picked mon.noname-a con 0x7f17240a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17240a9bd0 0x7f172413aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): picked mon.noname-c con 0x7f17240a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f172413f580 0x7f17240ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): picked mon.noname-b con 0x7f172413f580 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17240abde0 con 0x7f17240a8970 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172413bcb0 con 0x7f172413f580 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17240ac000 con 0x7f17240a9bd0 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f172b71d640 10 monclient(hunting): authenticate will time out at 1324.516523s 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f1729cba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f172413f580 0x7f17240ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f1728cb8640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17240a9bd0 0x7f172413aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.203+0000 7f17294b9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17240a8970 0x7f172413b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f172413f580 0x7f17240ac700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58854/0 (socket says 172.21.15.184:58854) 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 1 -- 172.21.15.184:0/2298634726 learned_addr learned my addr 172.21.15.184:0/2298634726 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 10 monclient(hunting): get_auth_request con 0x7f172413f580 auth_method 0 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1728cb8640 10 monclient(hunting): get_auth_request con 0x7f17240a9bd0 auth_method 0 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1728cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1728cb8640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1728cb8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f17294b9640 10 monclient(hunting): get_auth_request con 0x7f17240a8970 auth_method 0 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f17294b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f17294b9640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f17294b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1729cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1728cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1728cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f1728cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f17294b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f17294b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.204+0000 7f17294b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 1 -- 172.21.15.184:0/2298634726 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17240a9bd0 msgr2=0x7f172413aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 1 --2- 172.21.15.184:0/2298634726 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17240a9bd0 0x7f172413aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 1 -- 172.21.15.184:0/2298634726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17240a8970 msgr2=0x7f172413b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 1 --2- 172.21.15.184:0/2298634726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17240a8970 0x7f172413b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 10 monclient: _finish_hunting 0 2023-12-07T23:45:19.220 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 1 monclient: found mon.noname-b 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.207+0000 7f1729cba640 1 -- 172.21.15.184:0/2298634726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17200094d0 con 0x7f172413f580 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.209+0000 7f1729cba640 1 --2- 172.21.15.184:0/2298634726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f172413f580 0x7f17240ac700 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f171c007d10 tx=0x7f171c00e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.209+0000 7f1719ffb640 10 client.?.objecter ms_handle_connect 0x7f172413f580 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.209+0000 7f1719ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 1 -- 172.21.15.184:0/2298634726 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c014420 con 0x7f172413f580 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: dump: 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.221 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: _finish_auth 0 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 1 -- 172.21.15.184:0/2298634726 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171c0145a0 con 0x7f172413f580 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 5 monclient: authenticate success, global_id 4129 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 10 monclient: _renew_subs 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17240ac2e0 con 0x7f172413f580 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 1 -- 172.21.15.184:0/2298634726 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c014950 con 0x7f172413f580 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f1719ffb640 10 monclient: dump: 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f171b7fe640 4 set_mon_vals no callback set 2023-12-07T23:45:19.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 10 monclient: _renew_subs 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17240acc40 con 0x7f172413f580 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.212+0000 7f172b71d640 1 librados: init done 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.214+0000 7f172b71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.214+0000 7f172b71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.214+0000 7f172b71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.214+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172404e840 con 0x7f172413f580 2023-12-07T23:45:19.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.216+0000 7f1719ffb640 1 -- 172.21.15.184:0/2298634726 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f171c01e460 con 0x7f172413f580 2023-12-07T23:45:19.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.216+0000 7f1719ffb640 1 -- 172.21.15.184:0/2298634726 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f171c014e40 con 0x7f172413f580 2023-12-07T23:45:19.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.216+0000 7f1719ffb640 10 client.4129.objecter ms_dispatch 0x7f1724000b90 osd_map(4..4 src has 1..4) v4 2023-12-07T23:45:19.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.216+0000 7f1719ffb640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2023-12-07T23:45:19.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.216+0000 7f1719ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2023-12-07T23:45:19.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.216+0000 7f1719ffb640 20 client.4129.objecter dump_active .. 0 homeless 2023-12-07T23:45:19.225 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.223+0000 7f57d6813040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-07T23:45:19.231 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.229+0000 7f6055e4b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-07T23:45:19.272 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.269+0000 7f57d6813040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-07T23:45:19.317 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.314+0000 7f1719ffb640 1 -- 172.21.15.184:0/2298634726 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f171c01e710 con 0x7f172413f580 2023-12-07T23:45:19.317 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.314+0000 7f1719ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:19.317 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.314+0000 7f1719ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:19.349 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.347+0000 7f6055e4b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-07T23:45:19.371 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.368+0000 7f57d6813040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-07T23:45:19.389 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.386+0000 7f172b71d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b8306026-161e-40a0-bf4b-856ecf0b4d17", "id": 2}] 2023-12-07T23:45:19.389 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.386+0000 7f172b71d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b8306026-161e-40a0-bf4b-856ecf0b4d17", "id": 2}] 2023-12-07T23:45:19.389 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.386+0000 7f172b71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.389 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.386+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b8306026-161e-40a0-bf4b-856ecf0b4d17", "id": 2} v 0) v1 -- 0x7f17240725e0 con 0x7f172413f580 2023-12-07T23:45:19.394 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.392+0000 7f6055e4b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-07T23:45:19.461 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.458+0000 7f1719ffb640 1 -- 172.21.15.184:0/2298634726 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b8306026-161e-40a0-bf4b-856ecf0b4d17", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f171c01ea80 con 0x7f172413f580 2023-12-07T23:45:19.461 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.458+0000 7f1719ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b8306026-161e-40a0-bf4b-856ecf0b4d17", "id": 2}] 2023-12-07T23:45:19.461 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.458+0000 7f1719ffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:19.461 INFO:teuthology.orchestra.run.smithi184.stdout:2 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 10 librados: watch_flush enter 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 10 librados: watch_flush exit 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 20 client.4129.objecter shutdown clearing up homeless session... 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 10 client.4129.objecter successfully canceled tick 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 10 monclient: shutdown 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f172413f580 msgr2=0x7f17240ac700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.459+0000 7f172b71d640 1 --2- 172.21.15.184:0/2298634726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f172413f580 0x7f17240ac700 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f171c007d10 tx=0x7f171c00e4b0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 shutdown_connections 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 --2- 172.21.15.184:0/2298634726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f172413f580 0x7f17240ac700 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.463 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 --2- 172.21.15.184:0/2298634726 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17240a9bd0 0x7f172413aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.464 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 --2- 172.21.15.184:0/2298634726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17240a8970 0x7f172413b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.464 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 >> 172.21.15.184:0/2298634726 conn(0x7f1724066dd0 msgr2=0x7f17240b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:19.464 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 shutdown_connections 2023-12-07T23:45:19.464 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 -- 172.21.15.184:0/2298634726 wait complete. 2023-12-07T23:45:19.464 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.460+0000 7f172b71d640 1 librados: shutdown 2023-12-07T23:45:19.468 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.465+0000 7f57d6813040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-07T23:45:19.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19 3 2023-12-07T23:45:19.508 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.506+0000 7f6055e4b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-07T23:45:19.572 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.569+0000 7f16731fc640 10 monclient: get_monmap_and_config 2023-12-07T23:45:19.572 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.569+0000 7f16731fc640 10 monclient: build_initial_monmap 2023-12-07T23:45:19.572 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.569+0000 7f16731fc640 10 monclient: monmap: 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:19.570733+0000 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:19.570733+0000 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:19.573 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.570+0000 7f16731fc640 1 Processor -- start 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.570+0000 7f16731fc640 1 -- start start 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.570+0000 7f16731fc640 10 monclient: init 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c0a8970 0x7f166c13ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): picked mon.noname-b con 0x7f166c0a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f166c0a9bd0 0x7f166c13f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): picked mon.noname-a con 0x7f166c0a9bd0 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f166c1421a0 0x7f166c142630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): picked mon.noname-c con 0x7f166c1421a0 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c0aa950 con 0x7f166c0a9bd0 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c0a90e0 con 0x7f166c1421a0 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0aabc0 con 0x7f166c0a8970 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f16731fc640 10 monclient(hunting): authenticate will time out at 1324.884536s 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.571+0000 7f1671799640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f166c1421a0 0x7f166c142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1670f98640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c0a8970 0x7f166c13ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f166c0a9bd0 0x7f166c13f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f166c1421a0 0x7f166c142630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58862/0 (socket says 172.21.15.184:58862) 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 1 -- 172.21.15.184:0/2237986798 learned_addr learned my addr 172.21.15.184:0/2237986798 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 10 monclient(hunting): get_auth_request con 0x7f166c1421a0 auth_method 0 2023-12-07T23:45:19.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 10 monclient(hunting): get_auth_request con 0x7f166c0a9bd0 auth_method 0 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1670f98640 10 monclient(hunting): get_auth_request con 0x7f166c0a8970 auth_method 0 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1670f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1670f98640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1670f98640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1671799640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.579 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.579 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f166bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.579 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1670f98640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.579 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.572+0000 7f1670f98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.579 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.573+0000 7f1670f98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.579 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 1 -- 172.21.15.184:0/2237986798 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c0a8970 msgr2=0x7f166c13ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 1 --2- 172.21.15.184:0/2237986798 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c0a8970 0x7f166c13ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 1 -- 172.21.15.184:0/2237986798 >> v2:172.21.15.129:3300/0 conn(0x7f166c0a9bd0 msgr2=0x7f166c13f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 1 --2- 172.21.15.184:0/2237986798 >> v2:172.21.15.129:3300/0 conn(0x7f166c0a9bd0 0x7f166c13f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 10 monclient: _finish_hunting 0 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 1 monclient: found mon.noname-c 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.574+0000 7f1671799640 1 -- 172.21.15.184:0/2237986798 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c0aaea0 con 0x7f166c1421a0 2023-12-07T23:45:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.576+0000 7f1671799640 1 --2- 172.21.15.184:0/2237986798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1421a0 0x7f166c142630 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f166c0b5080 tx=0x7f166402bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e1ac4122612e7da server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:19.588 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 1 -- 172.21.15.184:0/2237986798 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166402c950 con 0x7f166c1421a0 2023-12-07T23:45:19.588 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.588 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:19.588 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: dump: 2023-12-07T23:45:19.588 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.589 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: _finish_auth 0 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: _check_auth_tickets 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 1 -- 172.21.15.184:0/2237986798 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166402cab0 con 0x7f166c1421a0 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 1 -- 172.21.15.184:0/2237986798 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166402ce40 con 0x7f166c1421a0 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f16731fc640 5 monclient: authenticate success, global_id 4132 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f16731fc640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f16731fc640 4 set_mon_vals no callback set 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.579+0000 7f166b7fe640 10 monclient: dump: 2023-12-07T23:45:19.590 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 10 monclient: shutdown 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 -- 172.21.15.184:0/2237986798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1421a0 msgr2=0x7f166c142630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.591 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 --2- 172.21.15.184:0/2237986798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1421a0 0x7f166c142630 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f166c0b5080 tx=0x7f166402bdb0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 -- 172.21.15.184:0/2237986798 shutdown_connections 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 --2- 172.21.15.184:0/2237986798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1421a0 0x7f166c142630 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 --2- 172.21.15.184:0/2237986798 >> v2:172.21.15.129:3300/0 conn(0x7f166c0a9bd0 0x7f166c13f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 --2- 172.21.15.184:0/2237986798 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c0a8970 0x7f166c13ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 -- 172.21.15.184:0/2237986798 >> 172.21.15.184:0/2237986798 conn(0x7f166c066dd0 msgr2=0x7f166c0b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 -- 172.21.15.184:0/2237986798 shutdown_connections 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.580+0000 7f16731fc640 1 -- 172.21.15.184:0/2237986798 wait complete. 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 10 monclient: build_initial_monmap 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 10 monclient: monmap: 2023-12-07T23:45:19.592 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:19.593 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 1 librados: starting msgr at 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 1 librados: starting objecter 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 1 Processor -- start 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 1 -- start start 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 1 librados: setting wanted keys 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 1 librados: calling monclient init 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.581+0000 7f16731fc640 10 monclient: init 2023-12-07T23:45:19.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:19.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:45:19.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c0a8970 0x7f166c13e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): picked mon.noname-b con 0x7f166c0a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:19.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c0a9bd0 0x7f166c1da780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): picked mon.noname-a con 0x7f166c0a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:19.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c1421a0 0x7f166c0b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): picked mon.noname-c con 0x7f166c1421a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c1451f0 con 0x7f166c0a9bd0 2023-12-07T23:45:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c1dacc0 con 0x7f166c0a8970 2023-12-07T23:45:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b6df0 con 0x7f166c1421a0 2023-12-07T23:45:19.597 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:19.597 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.582+0000 7f16731fc640 10 monclient(hunting): authenticate will time out at 1324.895879s 2023-12-07T23:45:19.597 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f166bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c0a9bd0 0x7f166c1da780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.597 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1670f98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c0a8970 0x7f166c13e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.597 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1671799640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c1421a0 0x7f166c0b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:19.597 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1671799640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c1421a0 0x7f166c0b5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.184:44752/0 (socket says 172.21.15.184:44752) 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1670f98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c0a8970 0x7f166c13e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58876/0 (socket says 172.21.15.184:58876) 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1671799640 1 -- 172.21.15.184:0/1156984694 learned_addr learned my addr 172.21.15.184:0/1156984694 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1671799640 10 monclient(hunting): get_auth_request con 0x7f166c1421a0 auth_method 0 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1671799640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1671799640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1671799640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1670f98640 10 monclient(hunting): get_auth_request con 0x7f166c0a8970 auth_method 0 2023-12-07T23:45:19.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1670f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1670f98640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f1670f98640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f166bfff640 10 monclient(hunting): get_auth_request con 0x7f166c0a9bd0 auth_method 0 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f166bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f166bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.583+0000 7f166bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f1671799640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f1671799640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f1671799640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f1670f98640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f1670f98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f1670f98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f166bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:19.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f166bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:19.600 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.584+0000 7f166bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 1 -- 172.21.15.184:0/1156984694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c1421a0 msgr2=0x7f166c0b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 1 --2- 172.21.15.184:0/1156984694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c1421a0 0x7f166c0b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 1 -- 172.21.15.184:0/1156984694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c0a9bd0 msgr2=0x7f166c1da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 1 --2- 172.21.15.184:0/1156984694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c0a9bd0 0x7f166c1da780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 10 monclient: _finish_hunting 0 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 1 monclient: found mon.noname-b 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.586+0000 7f1670f98640 1 -- 172.21.15.184:0/1156984694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664009b30 con 0x7f166c0a8970 2023-12-07T23:45:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.588+0000 7f1670f98640 1 --2- 172.21.15.184:0/1156984694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c0a8970 0x7f166c13e140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f166000e850 tx=0x7f166000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.588+0000 7f16697fa640 10 client.?.objecter ms_handle_connect 0x7f166c0a8970 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.588+0000 7f16697fa640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.590+0000 7f16697fa640 1 -- 172.21.15.184:0/1156984694 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16600095b0 con 0x7f166c0a8970 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.590+0000 7f16697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.590+0000 7f16697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.590+0000 7f16697fa640 10 monclient: dump: 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.604 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.590+0000 7f16697fa640 10 monclient: _finish_auth 0 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.590+0000 7f16697fa640 10 monclient: _check_auth_tickets 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 5 monclient: authenticate success, global_id 4135 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 10 monclient: _renew_subs 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c0b70d0 con 0x7f166c0a8970 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 10 monclient: _renew_subs 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c144e80 con 0x7f166c0a8970 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.591+0000 7f16731fc640 1 librados: init done 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.592+0000 7f16731fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.592+0000 7f16731fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:19.605 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.592+0000 7f16731fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.592+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c0aa350 con 0x7f166c0a8970 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.593+0000 7f16697fa640 1 -- 172.21.15.184:0/1156984694 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1660005990 con 0x7f166c0a8970 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.593+0000 7f16697fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.593+0000 7f16697fa640 1 -- 172.21.15.184:0/1156984694 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660005d00 con 0x7f166c0a8970 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.593+0000 7f16697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.593+0000 7f16697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.593+0000 7f16697fa640 10 monclient: dump: 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.606 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:19.607 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:19.607 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:19.607 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.607 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.593+0000 7f166affd640 4 set_mon_vals no callback set 2023-12-07T23:45:19.607 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.601+0000 7f6055e4b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-07T23:45:19.630 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.627+0000 7f57d6813040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-07T23:45:19.673 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.671+0000 7f57d6813040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-07T23:45:19.707 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 1 -- 172.21.15.184:0/1156984694 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f166000cca0 con 0x7f166c0a8970 2023-12-07T23:45:19.708 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 1 -- 172.21.15.184:0/1156984694 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f1660006870 con 0x7f166c0a8970 2023-12-07T23:45:19.708 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 10 client.4135.objecter ms_dispatch 0x7f166c000b90 osd_map(5..5 src has 1..5) v4 2023-12-07T23:45:19.708 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2023-12-07T23:45:19.708 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2023-12-07T23:45:19.708 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 20 client.4135.objecter dump_active .. 0 homeless 2023-12-07T23:45:19.708 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 1 -- 172.21.15.184:0/1156984694 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1660021060 con 0x7f166c0a8970 2023-12-07T23:45:19.709 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:19.709 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.704+0000 7f16697fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:19.754 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.752+0000 7f6055e4b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-07T23:45:19.800 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.798+0000 7f6055e4b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-07T23:45:19.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.801+0000 7f16731fc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19", "id": 3}] 2023-12-07T23:45:19.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.801+0000 7f16731fc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19", "id": 3}] 2023-12-07T23:45:19.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.801+0000 7f16731fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:19.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.801+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19", "id": 3} v 0) v1 -- 0x7f166c04e840 con 0x7f166c0a8970 2023-12-07T23:45:19.839 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:19.836+0000 7f57d6813040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-07T23:45:19.883 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.880+0000 7f16697fa640 1 -- 172.21.15.184:0/1156984694 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1660017070 con 0x7f166c0a8970 2023-12-07T23:45:19.884 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.881+0000 7f16697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19", "id": 3}] 2023-12-07T23:45:19.884 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.881+0000 7f16697fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:19.884 INFO:teuthology.orchestra.run.smithi184.stdout:3 2023-12-07T23:45:19.886 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.881+0000 7f16731fc640 10 librados: watch_flush enter 2023-12-07T23:45:19.886 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.881+0000 7f16731fc640 10 librados: watch_flush exit 2023-12-07T23:45:19.886 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.881+0000 7f16731fc640 20 client.4135.objecter shutdown clearing up homeless session... 2023-12-07T23:45:19.886 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.881+0000 7f16731fc640 10 client.4135.objecter successfully canceled tick 2023-12-07T23:45:19.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 10 monclient: shutdown 2023-12-07T23:45:19.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:19.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c0a8970 msgr2=0x7f166c13e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:19.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 --2- 172.21.15.184:0/1156984694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c0a8970 0x7f166c13e140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f166000e850 tx=0x7f166000ed10 comp rx=0 tx=0).stop 2023-12-07T23:45:19.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 shutdown_connections 2023-12-07T23:45:19.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 --2- 172.21.15.184:0/1156984694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f166c1421a0 0x7f166c0b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 --2- 172.21.15.184:0/1156984694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c0a9bd0 0x7f166c1da780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.888 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 --2- 172.21.15.184:0/1156984694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c0a8970 0x7f166c13e140 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:19.888 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 >> 172.21.15.184:0/1156984694 conn(0x7f166c066dd0 msgr2=0x7f166c13b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:19.888 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 shutdown_connections 2023-12-07T23:45:19.888 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 -- 172.21.15.184:0/1156984694 wait complete. 2023-12-07T23:45:19.888 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.882+0000 7f16731fc640 1 librados: shutdown 2023-12-07T23:45:19.896 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new ce9d5aa9-cc19-4fb5-8089-5e185304551e 4 2023-12-07T23:45:19.956 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:19.954+0000 7f6055e4b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-07T23:45:19.996 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.993+0000 7fcb7557d640 10 monclient: get_monmap_and_config 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.993+0000 7fcb7557d640 10 monclient: build_initial_monmap 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.993+0000 7fcb7557d640 10 monclient: monmap: 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:19.994825+0000 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:19.994825+0000 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:19.998 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.994+0000 7fcb7557d640 1 Processor -- start 2023-12-07T23:45:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.994+0000 7fcb7557d640 1 -- start start 2023-12-07T23:45:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.994+0000 7fcb7557d640 10 monclient: init 2023-12-07T23:45:19.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:19.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-12-07T23:45:19.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb700a8970 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:19.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcb700a9bd0 0x7fcb701421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:19.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb700a9bd0 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb7013f580 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb700aa950 con 0x7fcb700a8970 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb700a90e0 con 0x7fcb7013f580 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb70142df0 con 0x7fcb700a9bd0 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb7557d640 10 monclient(hunting): authenticate will time out at 1325.308490s 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb6e7fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcb700a9bd0 0x7fcb701421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb6effd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fcb700a8970 0x7fcb7013ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.184:50922/0 (socket says 172.21.15.184:50922) 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.995+0000 7fcb6effd640 1 -- 172.21.15.184:0/2918697295 learned_addr learned my addr 172.21.15.184:0/2918697295 (peer_addr_for_me v1:172.21.15.184:0/0) 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.996+0000 7fcb6e7fc640 10 monclient(hunting): get_auth_request con 0x7fcb700a9bd0 auth_method 0 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.996+0000 7fcb6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.996+0000 7fcb6e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.996+0000 7fcb6e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.996+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.996+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.001 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.996+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 1 -- 172.21.15.184:0/2918697295 >> v1:172.21.15.184:6789/0 conn(0x7fcb7013f580 legacy=0x7fcb701426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 1 -- 172.21.15.184:0/2918697295 >> v1:172.21.15.129:6789/0 conn(0x7fcb700a8970 legacy=0x7fcb7013ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 10 monclient: _finish_hunting 0 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 1 monclient: found mon.noname-c 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:19.998+0000 7fcb6e7fc640 1 -- 172.21.15.184:0/2918697295 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb700aabc0 con 0x7fcb700a9bd0 2023-12-07T23:45:20.003 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.000+0000 7fcb6e7fc640 1 --2- 172.21.15.184:0/2918697295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb701421a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcb600097a0 tx=0x7fcb6002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=769e53b9ce9e23af server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 1 -- 172.21.15.184:0/2918697295 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6002f050 con 0x7fcb700a9bd0 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: dump: 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.007 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: _finish_auth 0 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: _check_auth_tickets 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 1 -- 172.21.15.184:0/2918697295 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb60031700 con 0x7fcb700a9bd0 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb7557d640 5 monclient: authenticate success, global_id 4138 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb7557d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb7557d640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:20.008 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 1 -- 172.21.15.184:0/2918697295 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb60031a90 con 0x7fcb700a9bd0 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb7557d640 4 set_mon_vals no callback set 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.004+0000 7fcb6dffb640 10 monclient: dump: 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.009 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.012 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 10 monclient: shutdown 2023-12-07T23:45:20.012 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:20.012 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 1 -- 172.21.15.184:0/2918697295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 msgr2=0x7fcb701421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 1 --2- 172.21.15.184:0/2918697295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb701421a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcb600097a0 tx=0x7fcb6002b7a0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 1 -- 172.21.15.184:0/2918697295 shutdown_connections 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 1 --2- 172.21.15.184:0/2918697295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb701421a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 1 -- 172.21.15.184:0/2918697295 >> 172.21.15.184:0/2918697295 conn(0x7fcb70066dd0 msgr2=0x7fcb700b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 1 -- 172.21.15.184:0/2918697295 shutdown_connections 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.005+0000 7fcb7557d640 1 -- 172.21.15.184:0/2918697295 wait complete. 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 10 monclient: build_initial_monmap 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 10 monclient: monmap: 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.013 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 1 librados: starting msgr at 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 1 librados: starting objecter 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 1 Processor -- start 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 1 -- start start 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 1 librados: setting wanted keys 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 1 librados: calling monclient init 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.006+0000 7fcb7557d640 10 monclient: init 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb700a8970 0x7fcb7013a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb700a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:20.014 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb70138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb700a9bd0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcb7013f580 0x7fcb700ab350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb7013f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7013ae00 con 0x7fcb700a8970 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7013b850 con 0x7fcb700a9bd0 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7013ab00 con 0x7fcb7013f580 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb7557d640 10 monclient(hunting): authenticate will time out at 1325.320714s 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.007+0000 7fcb6e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb70138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6f7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcb7013f580 0x7fcb700ab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb700a8970 0x7fcb7013a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb70138220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58898/0 (socket says 172.21.15.184:58898) 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 1 -- 172.21.15.184:0/3578971342 learned_addr learned my addr 172.21.15.184:0/3578971342 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 10 monclient(hunting): get_auth_request con 0x7fcb700a9bd0 auth_method 0 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 10 monclient(hunting): get_auth_request con 0x7fcb700a8970 auth_method 0 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6f7fe640 10 monclient(hunting): get_auth_request con 0x7fcb7013f580 auth_method 0 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.016 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.008+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.009+0000 7fcb6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.009+0000 7fcb6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.009+0000 7fcb6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 1 -- 172.21.15.184:0/3578971342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcb7013f580 msgr2=0x7fcb700ab350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 1 --2- 172.21.15.184:0/3578971342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcb7013f580 0x7fcb700ab350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 1 -- 172.21.15.184:0/3578971342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb700a8970 msgr2=0x7fcb7013a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 1 --2- 172.21.15.184:0/3578971342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb700a8970 0x7fcb7013a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 10 monclient: _finish_hunting 0 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 1 monclient: found mon.noname-b 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.011+0000 7fcb6e7fc640 1 -- 172.21.15.184:0/3578971342 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb60009400 con 0x7fcb700a9bd0 2023-12-07T23:45:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.013+0000 7fcb6e7fc640 1 --2- 172.21.15.184:0/3578971342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb70138220 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb6002bdb0 tx=0x7fcb600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:20.019 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.013+0000 7fcb677fe640 10 client.?.objecter ms_handle_connect 0x7fcb700a9bd0 2023-12-07T23:45:20.019 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.013+0000 7fcb677fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:20.020 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 1 -- 172.21.15.184:0/3578971342 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb60033c70 con 0x7fcb700a9bd0 2023-12-07T23:45:20.020 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: dump: 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: _finish_auth 0 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: _check_auth_tickets 2023-12-07T23:45:20.021 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 5 monclient: authenticate success, global_id 4141 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 1 -- 172.21.15.184:0/3578971342 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb60033df0 con 0x7fcb700a9bd0 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 10 monclient: _renew_subs 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb70138760 con 0x7fcb700a9bd0 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 1 -- 172.21.15.184:0/3578971342 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6003e6c0 con 0x7fcb700a9bd0 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb677fe640 10 monclient: dump: 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.022 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb6d7fa640 4 set_mon_vals no callback set 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 10 monclient: _renew_subs 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7013bda0 con 0x7fcb700a9bd0 2023-12-07T23:45:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.017+0000 7fcb7557d640 1 librados: init done 2023-12-07T23:45:20.024 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.018+0000 7fcb7557d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.024 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.018+0000 7fcb7557d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.024 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.018+0000 7fcb7557d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.024 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.018+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7013afb0 con 0x7fcb700a9bd0 2023-12-07T23:45:20.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 1 -- 172.21.15.184:0/3578971342 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcb6003a040 con 0x7fcb700a9bd0 2023-12-07T23:45:20.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 1 -- 172.21.15.184:0/3578971342 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fcb60053940 con 0x7fcb700a9bd0 2023-12-07T23:45:20.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 10 client.4141.objecter ms_dispatch 0x7fcb70000b90 osd_map(6..6 src has 1..6) v4 2023-12-07T23:45:20.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2023-12-07T23:45:20.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2023-12-07T23:45:20.120 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 20 client.4141.objecter dump_active .. 0 homeless 2023-12-07T23:45:20.120 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 1 -- 172.21.15.184:0/3578971342 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fcb7013afb0 con 0x7fcb700a9bd0 2023-12-07T23:45:20.120 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.120 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.116+0000 7fcb677fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.186+0000 7fcb7557d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ce9d5aa9-cc19-4fb5-8089-5e185304551e", "id": 4}] 2023-12-07T23:45:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.186+0000 7fcb7557d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ce9d5aa9-cc19-4fb5-8089-5e185304551e", "id": 4}] 2023-12-07T23:45:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.186+0000 7fcb7557d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.186+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ce9d5aa9-cc19-4fb5-8089-5e185304551e", "id": 4} v 0) v1 -- 0x7fcb7004e840 con 0x7fcb700a9bd0 2023-12-07T23:45:20.278 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.275+0000 7fcb677fe640 1 -- 172.21.15.184:0/3578971342 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce9d5aa9-cc19-4fb5-8089-5e185304551e", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcb60044070 con 0x7fcb700a9bd0 2023-12-07T23:45:20.278 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.275+0000 7fcb677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ce9d5aa9-cc19-4fb5-8089-5e185304551e", "id": 4}] 2023-12-07T23:45:20.278 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.275+0000 7fcb677fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:20.278 INFO:teuthology.orchestra.run.smithi184.stdout:4 2023-12-07T23:45:20.279 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 10 librados: watch_flush enter 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 10 librados: watch_flush exit 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 20 client.4141.objecter shutdown clearing up homeless session... 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 10 client.4141.objecter successfully canceled tick 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 10 monclient: shutdown 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 msgr2=0x7fcb70138220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 --2- 172.21.15.184:0/3578971342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb70138220 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb6002bdb0 tx=0x7fcb600046c0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 shutdown_connections 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 --2- 172.21.15.184:0/3578971342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcb7013f580 0x7fcb700ab350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 --2- 172.21.15.184:0/3578971342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb700a9bd0 0x7fcb70138220 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 --2- 172.21.15.184:0/3578971342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb700a8970 0x7fcb7013a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 >> 172.21.15.184:0/3578971342 conn(0x7fcb70066dd0 msgr2=0x7fcb701421a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 shutdown_connections 2023-12-07T23:45:20.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 -- 172.21.15.184:0/3578971342 wait complete. 2023-12-07T23:45:20.281 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.276+0000 7fcb7557d640 1 librados: shutdown 2023-12-07T23:45:20.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 714c8fce-fdd4-49b3-9cce-fd43ca74b16d 5 2023-12-07T23:45:20.387 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.384+0000 7f81a9a82640 10 monclient: get_monmap_and_config 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.384+0000 7f81a9a82640 10 monclient: build_initial_monmap 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.384+0000 7f81a9a82640 10 monclient: monmap: 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:20.385723+0000 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:20.385723+0000 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:20.388 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:20.389 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:20.389 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:20.389 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.385+0000 7f81a9a82640 1 Processor -- start 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.385+0000 7f81a9a82640 1 -- start start 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.385+0000 7f81a9a82640 10 monclient: init 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81a40a8970 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81a40a9bd0 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:20.391 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a41428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient(hunting): picked mon.noname-b con 0x7f81a413f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a40aa950 con 0x7f81a40a9bd0 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a40a90e0 con 0x7f81a40a8970 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4142df0 con 0x7f81a413f580 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.386+0000 7f81a9a82640 10 monclient(hunting): authenticate will time out at 1325.699704s 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.387+0000 7f81a37fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f81a40a8970 0x7f81a413ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:60962/0 (socket says 172.21.15.184:60962) 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.387+0000 7f81a37fe640 1 -- 172.21.15.184:0/2467560494 learned_addr learned my addr 172.21.15.184:0/2467560494 (peer_addr_for_me v1:172.21.15.184:0/0) 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.387+0000 7f81a3fff640 1 --2- 172.21.15.184:0/2467560494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a41428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.387+0000 7f81a3fff640 10 monclient(hunting): get_auth_request con 0x7f81a413f580 auth_method 0 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.387+0000 7f81a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.387+0000 7f81a3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.387+0000 7f81a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.393 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.388+0000 7f81a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.393 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.388+0000 7f81a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.393 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.388+0000 7f81a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a27fc640 1 -- 172.21.15.184:0/2467560494 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792798590 0 0) 0x7f81a40a90e0 con 0x7f81a40a8970 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a27fc640 10 monclient(hunting): _negotiate 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a27fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a27fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a27fc640 10 monclient(hunting): my global_id is 4144 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a27fc640 1 -- 172.21.15.184:0/2467560494 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c003520 con 0x7f81a40a8970 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 10 monclient(hunting): handle_auth_done global_id 4130 payload 274 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 1 -- 172.21.15.184:0/2467560494 >> v1:172.21.15.184:6789/0 conn(0x7f81a40a8970 legacy=0x7f81a413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 1 -- 172.21.15.184:0/2467560494 >> v1:172.21.15.129:6789/0 conn(0x7f81a40a9bd0 legacy=0x7f81a41421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.395 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 10 monclient: _finish_hunting 0 2023-12-07T23:45:20.395 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 1 monclient: found mon.noname-b 2023-12-07T23:45:20.395 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:20.395 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:45:20.395 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.389+0000 7f81a3fff640 1 -- 172.21.15.184:0/2467560494 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a40aabc0 con 0x7f81a413f580 2023-12-07T23:45:20.395 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.391+0000 7f81a3fff640 1 --2- 172.21.15.184:0/2467560494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a41428b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f819c009dc0 tx=0x7f819c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=774eb4f14359a4ef server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:20.397 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 1 -- 172.21.15.184:0/2467560494 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c02cbb0 con 0x7f81a413f580 2023-12-07T23:45:20.398 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.398 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:20.398 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: dump: 2023-12-07T23:45:20.398 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.398 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.399 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: _finish_auth 0 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: _check_auth_tickets 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 1 -- 172.21.15.184:0/2467560494 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c02cd10 con 0x7f81a413f580 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 5 monclient: authenticate success, global_id 4130 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 4 set_mon_vals no callback set 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 1 -- 172.21.15.184:0/2467560494 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c034660 con 0x7f81a413f580 2023-12-07T23:45:20.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a27fc640 10 monclient: dump: 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.401 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.405 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 10 monclient: shutdown 2023-12-07T23:45:20.405 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 1 -- 172.21.15.184:0/2467560494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 msgr2=0x7f81a41428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 1 --2- 172.21.15.184:0/2467560494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a41428b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f819c009dc0 tx=0x7f819c02be00 comp rx=0 tx=0).stop 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 1 -- 172.21.15.184:0/2467560494 shutdown_connections 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.395+0000 7f81a9a82640 1 --2- 172.21.15.184:0/2467560494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a41428b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.396+0000 7f81a9a82640 1 -- 172.21.15.184:0/2467560494 >> 172.21.15.184:0/2467560494 conn(0x7f81a4066dd0 msgr2=0x7f81a40b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.396+0000 7f81a9a82640 1 -- 172.21.15.184:0/2467560494 shutdown_connections 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.396+0000 7f81a9a82640 1 -- 172.21.15.184:0/2467560494 wait complete. 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.396+0000 7f81a9a82640 10 monclient: build_initial_monmap 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.396+0000 7f81a9a82640 10 monclient: monmap: 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:20.406 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.396+0000 7f81a9a82640 1 librados: starting msgr at 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.396+0000 7f81a9a82640 1 librados: starting objecter 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 Processor -- start 2023-12-07T23:45:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 -- start start 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 librados: setting wanted keys 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 librados: calling monclient init 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient: init 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81a40a8970 0x7f81a40ac060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): picked mon.noname-b con 0x7f81a40a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a40a9bd0 0x7f81a413a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): picked mon.noname-a con 0x7f81a40a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a40adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): picked mon.noname-c con 0x7f81a413f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a40acbf0 con 0x7f81a40a9bd0 2023-12-07T23:45:20.408 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a40ac5a0 con 0x7f81a40a8970 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a413a850 con 0x7f81a413f580 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.397+0000 7f81a9a82640 10 monclient(hunting): authenticate will time out at 1325.711176s 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81a40a8970 0x7f81a40ac060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a3fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a40adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a40a9bd0 0x7f81a413a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81a40a8970 0x7f81a40ac060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58902/0 (socket says 172.21.15.184:58902) 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a37fe640 1 -- 172.21.15.184:0/1702840322 learned_addr learned my addr 172.21.15.184:0/1702840322 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a37fe640 10 monclient(hunting): get_auth_request con 0x7f81a40a8970 auth_method 0 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a2ffd640 10 monclient(hunting): get_auth_request con 0x7f81a40a9bd0 auth_method 0 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a3fff640 10 monclient(hunting): get_auth_request con 0x7f81a413f580 auth_method 0 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.398+0000 7f81a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.410 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.399+0000 7f81a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 1 -- 172.21.15.184:0/1702840322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 msgr2=0x7f81a40adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 1 --2- 172.21.15.184:0/1702840322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a40adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 1 -- 172.21.15.184:0/1702840322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a40a9bd0 msgr2=0x7f81a413a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 1 --2- 172.21.15.184:0/1702840322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a40a9bd0 0x7f81a413a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 10 monclient: _finish_hunting 0 2023-12-07T23:45:20.411 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 1 monclient: found mon.noname-b 2023-12-07T23:45:20.412 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:20.412 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.412 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.402+0000 7f81a37fe640 1 -- 172.21.15.184:0/1702840322 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c009a70 con 0x7f81a40a8970 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.404+0000 7f81a37fe640 1 --2- 172.21.15.184:0/1702840322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81a40a8970 0x7f81a40ac060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f819400c530 tx=0x7f819400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.404+0000 7f818bfff640 10 client.?.objecter ms_handle_connect 0x7f81a40a8970 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.404+0000 7f818bfff640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 1 -- 172.21.15.184:0/1702840322 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194005ce0 con 0x7f81a40a8970 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: dump: 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: _finish_auth 0 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: _check_auth_tickets 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:20.414 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 1 -- 172.21.15.184:0/1702840322 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8194005e40 con 0x7f81a40a8970 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 1 -- 172.21.15.184:0/1702840322 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81940072b0 con 0x7f81a40a8970 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 5 monclient: authenticate success, global_id 4147 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 10 monclient: _renew_subs 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a40ad630 con 0x7f81a40a8970 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a1ffb640 4 set_mon_vals no callback set 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f818bfff640 10 monclient: dump: 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 10 monclient: _renew_subs 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a40ac810 con 0x7f81a40a8970 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.408+0000 7f81a9a82640 1 librados: init done 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.409+0000 7f81a9a82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.409+0000 7f81a9a82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.416 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.409+0000 7f81a9a82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.417 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.409+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a40aca00 con 0x7f81a40a8970 2023-12-07T23:45:20.514 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 1 -- 172.21.15.184:0/1702840322 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f819401c440 con 0x7f81a40a8970 2023-12-07T23:45:20.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 1 -- 172.21.15.184:0/1702840322 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f819401b930 con 0x7f81a40a8970 2023-12-07T23:45:20.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 10 client.4147.objecter ms_dispatch 0x7f81a4000b90 osd_map(7..7 src has 1..7) v4 2023-12-07T23:45:20.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2023-12-07T23:45:20.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2023-12-07T23:45:20.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 20 client.4147.objecter dump_active .. 0 homeless 2023-12-07T23:45:20.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 1 -- 172.21.15.184:0/1702840322 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f8194017070 con 0x7f81a40a8970 2023-12-07T23:45:20.572 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.572 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.510+0000 7f818bfff640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:20.586 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.583+0000 7f81a9a82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "714c8fce-fdd4-49b3-9cce-fd43ca74b16d", "id": 5}] 2023-12-07T23:45:20.586 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.583+0000 7f81a9a82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "714c8fce-fdd4-49b3-9cce-fd43ca74b16d", "id": 5}] 2023-12-07T23:45:20.586 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.583+0000 7f81a9a82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.586 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.583+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "714c8fce-fdd4-49b3-9cce-fd43ca74b16d", "id": 5} v 0) v1 -- 0x7f81a404e840 con 0x7f81a40a8970 2023-12-07T23:45:20.653 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.651+0000 7f818bfff640 1 -- 172.21.15.184:0/1702840322 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "714c8fce-fdd4-49b3-9cce-fd43ca74b16d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f819401bc40 con 0x7f81a40a8970 2023-12-07T23:45:20.654 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.651+0000 7f818bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "714c8fce-fdd4-49b3-9cce-fd43ca74b16d", "id": 5}] 2023-12-07T23:45:20.654 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.651+0000 7f818bfff640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:20.654 INFO:teuthology.orchestra.run.smithi184.stdout:5 2023-12-07T23:45:20.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 10 librados: watch_flush enter 2023-12-07T23:45:20.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 10 librados: watch_flush exit 2023-12-07T23:45:20.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 20 client.4147.objecter shutdown clearing up homeless session... 2023-12-07T23:45:20.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 10 client.4147.objecter successfully canceled tick 2023-12-07T23:45:20.655 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 10 monclient: shutdown 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81a40a8970 msgr2=0x7f81a40ac060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 --2- 172.21.15.184:0/1702840322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81a40a8970 0x7f81a40ac060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f819400c530 tx=0x7f819400c9f0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 shutdown_connections 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 --2- 172.21.15.184:0/1702840322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81a413f580 0x7f81a40adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 --2- 172.21.15.184:0/1702840322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a40a9bd0 0x7f81a413a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 --2- 172.21.15.184:0/1702840322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81a40a8970 0x7f81a40ac060 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 >> 172.21.15.184:0/1702840322 conn(0x7f81a4066dd0 msgr2=0x7f81a40b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 shutdown_connections 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 -- 172.21.15.184:0/1702840322 wait complete. 2023-12-07T23:45:20.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.652+0000 7f81a9a82640 1 librados: shutdown 2023-12-07T23:45:20.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 17d9fe56-0780-437b-b6ab-43f48471716a 6 2023-12-07T23:45:20.666 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:20.664+0000 7f57d6813040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-07T23:45:20.777 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:20.775+0000 7f6055e4b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.775+0000 7f812c79e640 10 monclient: get_monmap_and_config 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.775+0000 7f812c79e640 10 monclient: build_initial_monmap 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.775+0000 7f812c79e640 10 monclient: monmap: 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:20.775978+0000 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:20.775978+0000 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:20.778 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:20.779 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:20.779 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:20.779 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:20.779 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.779 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.775+0000 7f812c79e640 1 Processor -- start 2023-12-07T23:45:20.779 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.775+0000 7f812c79e640 1 -- start start 2023-12-07T23:45:20.779 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.775+0000 7f812c79e640 10 monclient: init 2023-12-07T23:45:20.780 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:20.780 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-12-07T23:45:20.780 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81240a8970 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f81240a9bd0 0x7f81241421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient(hunting): picked mon.noname-c con 0x7f81240a9bd0 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f812413f580 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81240aa950 con 0x7f81240a8970 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81240a90e0 con 0x7f812413f580 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8124142df0 con 0x7f81240a9bd0 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:20.781 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.776+0000 7f812c79e640 10 monclient(hunting): authenticate will time out at 1326.089817s 2023-12-07T23:45:20.786 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.781+0000 7f812a53a640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f81240a8970 0x7f812413ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.184:50934/0 (socket says 172.21.15.184:50934) 2023-12-07T23:45:20.786 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.781+0000 7f812a53a640 1 -- 172.21.15.184:0/3374847329 learned_addr learned my addr 172.21.15.184:0/3374847329 (peer_addr_for_me v1:172.21.15.184:0/0) 2023-12-07T23:45:20.786 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.782+0000 7f8129d39640 1 --2- 172.21.15.184:0/3374847329 >> v2:172.21.15.184:3300/0 conn(0x7f81240a9bd0 0x7f81241421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.786 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.783+0000 7f8129d39640 10 monclient(hunting): get_auth_request con 0x7f81240a9bd0 auth_method 0 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.783+0000 7f8129d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.783+0000 7f8129d39640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.783+0000 7f8129d39640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.784+0000 7f8129538640 1 -- 172.21.15.184:0/3374847329 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849379303 0 0) 0x7f81240aa950 con 0x7f81240a8970 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.784+0000 7f8129538640 10 monclient(hunting): _negotiate 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.784+0000 7f8129538640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.784+0000 7f8129538640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.784+0000 7f8129538640 10 monclient(hunting): my global_id is 4131 2023-12-07T23:45:20.787 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.784+0000 7f8129538640 1 -- 172.21.15.184:0/3374847329 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8110003520 con 0x7f81240a8970 2023-12-07T23:45:20.788 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.785+0000 7f8129d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.788 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.785+0000 7f8129d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.788 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.785+0000 7f8129d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 1 -- 172.21.15.184:0/3374847329 >> v1:172.21.15.184:6789/0 conn(0x7f812413f580 legacy=0x7f81241426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 1 -- 172.21.15.184:0/3374847329 >> v1:172.21.15.129:6789/0 conn(0x7f81240a8970 legacy=0x7f812413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 10 monclient: _finish_hunting 0 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 1 monclient: found mon.noname-c 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.786+0000 7f8129d39640 1 -- 172.21.15.184:0/3374847329 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81240aabc0 con 0x7f81240a9bd0 2023-12-07T23:45:20.791 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.788+0000 7f8129d39640 1 --2- 172.21.15.184:0/3374847329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81240a9bd0 0x7f81241421a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f81140097a0 tx=0x7f811402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=570b275bde5fdb2 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 1 -- 172.21.15.184:0/3374847329 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811402f050 con 0x7f81240a9bd0 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: dump: 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.802 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: _finish_auth 0 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: _check_auth_tickets 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 1 -- 172.21.15.184:0/3374847329 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8114031700 con 0x7f81240a9bd0 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 5 monclient: authenticate success, global_id 4150 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 1 -- 172.21.15.184:0/3374847329 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8114031a90 con 0x7f81240a9bd0 2023-12-07T23:45:20.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 4 set_mon_vals no callback set 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f8129538640 10 monclient: dump: 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 10 monclient: shutdown 2023-12-07T23:45:20.804 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 1 -- 172.21.15.184:0/3374847329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81240a9bd0 msgr2=0x7f81241421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 1 --2- 172.21.15.184:0/3374847329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81240a9bd0 0x7f81241421a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f81140097a0 tx=0x7f811402b7a0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 1 -- 172.21.15.184:0/3374847329 shutdown_connections 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 1 --2- 172.21.15.184:0/3374847329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81240a9bd0 0x7f81241421a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.793+0000 7f812c79e640 1 -- 172.21.15.184:0/3374847329 >> 172.21.15.184:0/3374847329 conn(0x7f8124066dd0 msgr2=0x7f81240b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.794+0000 7f812c79e640 1 -- 172.21.15.184:0/3374847329 shutdown_connections 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.794+0000 7f812c79e640 1 -- 172.21.15.184:0/3374847329 wait complete. 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.794+0000 7f812c79e640 10 monclient: build_initial_monmap 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.794+0000 7f812c79e640 10 monclient: monmap: 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:20.805 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.794+0000 7f812c79e640 1 librados: starting msgr at 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.794+0000 7f812c79e640 1 librados: starting objecter 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 Processor -- start 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 -- start start 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 librados: setting wanted keys 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 librados: calling monclient init 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient: init 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T23:45:20.806 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81240a8970 0x7f81240ac190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): picked mon.noname-a con 0x7f81240a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81240a9bd0 0x7f81240ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): picked mon.noname-c con 0x7f81240a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812413f580 0x7f8124135800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): picked mon.noname-b con 0x7f812413f580 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81241393a0 con 0x7f81240a8970 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240acc10 con 0x7f812413f580 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81240adf10 con 0x7f81240a9bd0 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.795+0000 7f812c79e640 10 monclient(hunting): authenticate will time out at 1326.109002s 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812413f580 0x7f8124135800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.807 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812a53a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81240a8970 0x7f81240ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f8129d39640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81240a9bd0 0x7f81240ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812413f580 0x7f8124135800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58920/0 (socket says 172.21.15.184:58920) 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 1 -- 172.21.15.184:0/2932756175 learned_addr learned my addr 172.21.15.184:0/2932756175 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 10 monclient(hunting): get_auth_request con 0x7f812413f580 auth_method 0 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812a53a640 10 monclient(hunting): get_auth_request con 0x7f81240a8970 auth_method 0 2023-12-07T23:45:20.808 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812a53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812a53a640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812a53a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f8129d39640 10 monclient(hunting): get_auth_request con 0x7f81240a9bd0 auth_method 0 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f8129d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f8129d39640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f8129d39640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.809 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.796+0000 7f812ad3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.797+0000 7f812a53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.797+0000 7f812a53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.797+0000 7f812a53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.797+0000 7f8129d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.797+0000 7f8129d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.797+0000 7f8129d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2023-12-07T23:45:20.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 1 -- 172.21.15.184:0/2932756175 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81240a9bd0 msgr2=0x7f81240ac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 1 --2- 172.21.15.184:0/2932756175 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81240a9bd0 0x7f81240ac6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 1 -- 172.21.15.184:0/2932756175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81240a8970 msgr2=0x7f81240ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 1 --2- 172.21.15.184:0/2932756175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81240a8970 0x7f81240ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 10 monclient: _finish_hunting 0 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 1 monclient: found mon.noname-b 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.799+0000 7f812ad3b640 1 -- 172.21.15.184:0/2932756175 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8114009400 con 0x7f812413f580 2023-12-07T23:45:20.815 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.801+0000 7f812ad3b640 1 --2- 172.21.15.184:0/2932756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812413f580 0x7f8124135800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f811c00b9e0 tx=0x7f811c00bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:20.815 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.801+0000 7f811affd640 10 client.?.objecter ms_handle_connect 0x7f812413f580 2023-12-07T23:45:20.815 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.801+0000 7f811affd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:20.815 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 1 -- 172.21.15.184:0/2932756175 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c00caa0 con 0x7f812413f580 2023-12-07T23:45:20.815 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.815 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:20.816 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: dump: 2023-12-07T23:45:20.816 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.816 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.816 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.817 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.817 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.817 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.817 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.817 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.817 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: _finish_auth 0 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: _check_auth_tickets 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 1 -- 172.21.15.184:0/2932756175 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c00cc00 con 0x7f812413f580 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 5 monclient: authenticate success, global_id 4153 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 10 monclient: _renew_subs 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81240ace50 con 0x7f812413f580 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 1 -- 172.21.15.184:0/2932756175 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c014440 con 0x7f812413f580 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f811affd640 10 monclient: dump: 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f8128d37640 4 set_mon_vals no callback set 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 10 monclient: _renew_subs 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8124135d40 con 0x7f812413f580 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.805+0000 7f812c79e640 1 librados: init done 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.806+0000 7f812c79e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.806+0000 7f812c79e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.806+0000 7f812c79e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.806+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81241d6d80 con 0x7f812413f580 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.808+0000 7f811affd640 1 -- 172.21.15.184:0/2932756175 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f811c0148c0 con 0x7f812413f580 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.808+0000 7f811affd640 1 -- 172.21.15.184:0/2932756175 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f811c01d9a0 con 0x7f812413f580 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.808+0000 7f811affd640 10 client.4153.objecter ms_dispatch 0x7f8124000b90 osd_map(8..8 src has 1..8) v4 2023-12-07T23:45:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.808+0000 7f811affd640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2023-12-07T23:45:20.821 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.808+0000 7f811affd640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2023-12-07T23:45:20.821 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.808+0000 7f811affd640 20 client.4153.objecter dump_active .. 0 homeless 2023-12-07T23:45:20.907 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.905+0000 7f811affd640 1 -- 172.21.15.184:0/2932756175 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f81241d6d80 con 0x7f812413f580 2023-12-07T23:45:20.908 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.905+0000 7f811affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:20.908 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.905+0000 7f811affd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:20.969 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:20.966+0000 7f57d6813040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-07T23:45:20.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.996+0000 7f812c79e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "17d9fe56-0780-437b-b6ab-43f48471716a", "id": 6}] 2023-12-07T23:45:20.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.996+0000 7f812c79e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "17d9fe56-0780-437b-b6ab-43f48471716a", "id": 6}] 2023-12-07T23:45:20.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.996+0000 7f812c79e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:20.999 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:20.996+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "17d9fe56-0780-437b-b6ab-43f48471716a", "id": 6} v 0) v1 -- 0x7f812404e840 con 0x7f812413f580 2023-12-07T23:45:21.022 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:21.019+0000 7f57d6813040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-07T23:45:21.055 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:21.053+0000 7f6055e4b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-07T23:45:21.061 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.059+0000 7f811affd640 1 -- 172.21.15.184:0/2932756175 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17d9fe56-0780-437b-b6ab-43f48471716a", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f811c019070 con 0x7f812413f580 2023-12-07T23:45:21.061 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.059+0000 7f811affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "17d9fe56-0780-437b-b6ab-43f48471716a", "id": 6}] 2023-12-07T23:45:21.062 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.059+0000 7f811affd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:21.062 INFO:teuthology.orchestra.run.smithi184.stdout:6 2023-12-07T23:45:21.063 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 10 librados: watch_flush enter 2023-12-07T23:45:21.063 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 10 librados: watch_flush exit 2023-12-07T23:45:21.063 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 20 client.4153.objecter shutdown clearing up homeless session... 2023-12-07T23:45:21.063 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 10 client.4153.objecter successfully canceled tick 2023-12-07T23:45:21.063 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 10 monclient: shutdown 2023-12-07T23:45:21.063 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812413f580 msgr2=0x7f8124135800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 --2- 172.21.15.184:0/2932756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812413f580 0x7f8124135800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f811c00b9e0 tx=0x7f811c00bea0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 shutdown_connections 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 --2- 172.21.15.184:0/2932756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812413f580 0x7f8124135800 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 --2- 172.21.15.184:0/2932756175 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81240a9bd0 0x7f81240ac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 --2- 172.21.15.184:0/2932756175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81240a8970 0x7f81240ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 >> 172.21.15.184:0/2932756175 conn(0x7f8124066dd0 msgr2=0x7f81240b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 shutdown_connections 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 -- 172.21.15.184:0/2932756175 wait complete. 2023-12-07T23:45:21.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.060+0000 7f812c79e640 1 librados: shutdown 2023-12-07T23:45:21.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 29e1d98a-ea92-42d8-b827-0004493a6525 7 2023-12-07T23:45:21.108 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:21.106+0000 7f6055e4b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-07T23:45:21.134 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:21.131+0000 7f57d6813040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-07T23:45:21.175 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.172+0000 7f944cdf0640 10 monclient: get_monmap_and_config 2023-12-07T23:45:21.175 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.172+0000 7f944cdf0640 10 monclient: build_initial_monmap 2023-12-07T23:45:21.175 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.172+0000 7f944cdf0640 10 monclient: monmap: 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:45:21.173501+0000 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:45:21.173501+0000 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.173+0000 7f944cdf0640 1 Processor -- start 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.173+0000 7f944cdf0640 1 -- start start 2023-12-07T23:45:21.176 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.173+0000 7f944cdf0640 10 monclient: init 2023-12-07T23:45:21.178 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.173+0000 7f944cdf0640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:21.178 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.173+0000 7f944cdf0640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.173+0000 7f944cdf0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f94480a8970 0x7f944813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): picked mon.noname-a con 0x7f94480a8970 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f944813f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): picked mon.noname-b con 0x7f94480a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f94481421a0 0x7f9448142630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): picked mon.noname-c con 0x7f94481421a0 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94480aa950 con 0x7f94480a8970 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94480a90e0 con 0x7f94481421a0 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94480aabc0 con 0x7f94480a9bd0 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:21.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944cdf0640 10 monclient(hunting): authenticate will time out at 1326.487381s 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f944659c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f94480a8970 0x7f944813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.174+0000 7f9445d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f944813f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f944813f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.184:44782/0 (socket says 172.21.15.184:44782) 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 1 -- 172.21.15.184:0/2793857525 learned_addr learned my addr 172.21.15.184:0/2793857525 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 10 monclient(hunting): get_auth_request con 0x7f94480a9bd0 auth_method 0 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f944659c640 10 monclient(hunting): get_auth_request con 0x7f94480a8970 auth_method 0 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f944659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f944659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f944659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:21.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f9445d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:21.181 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f944659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:21.181 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f944659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:21.181 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.175+0000 7f944659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:21.181 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 1 -- 172.21.15.184:0/2793857525 >> v2:172.21.15.184:3300/0 conn(0x7f94481421a0 msgr2=0x7f9448142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 1 --2- 172.21.15.184:0/2793857525 >> v2:172.21.15.184:3300/0 conn(0x7f94481421a0 0x7f9448142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 1 -- 172.21.15.184:0/2793857525 >> v2:172.21.15.129:3300/0 conn(0x7f94480a8970 msgr2=0x7f944813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 1 --2- 172.21.15.184:0/2793857525 >> v2:172.21.15.129:3300/0 conn(0x7f94480a8970 0x7f944813ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 10 monclient: _finish_hunting 0 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 1 monclient: found mon.noname-b 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.177+0000 7f9445d9b640 1 -- 172.21.15.184:0/2793857525 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94480aaea0 con 0x7f94480a9bd0 2023-12-07T23:45:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.179+0000 7f9445d9b640 1 --2- 172.21.15.184:0/2793857525 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f944813f3b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f94480b5080 tx=0x7f943802b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d67eb09e550395d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:21.185 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 1 -- 172.21.15.184:0/2793857525 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438031750 con 0x7f94480a9bd0 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: dump: 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:21.186 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: _finish_auth 0 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: _check_auth_tickets 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 1 -- 172.21.15.184:0/2793857525 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94380318b0 con 0x7f94480a9bd0 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944cdf0640 5 monclient: authenticate success, global_id 4139 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944cdf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944cdf0640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944cdf0640 4 set_mon_vals no callback set 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 1 -- 172.21.15.184:0/2793857525 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438031c40 con 0x7f94480a9bd0 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.182+0000 7f944559a640 10 monclient: dump: 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:21.187 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 10 monclient: shutdown 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 -- 172.21.15.184:0/2793857525 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 msgr2=0x7f944813f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 --2- 172.21.15.184:0/2793857525 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f944813f3b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f94480b5080 tx=0x7f943802b7f0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.188 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 -- 172.21.15.184:0/2793857525 shutdown_connections 2023-12-07T23:45:21.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 --2- 172.21.15.184:0/2793857525 >> v2:172.21.15.184:3300/0 conn(0x7f94481421a0 0x7f9448142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 --2- 172.21.15.184:0/2793857525 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f944813f3b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 --2- 172.21.15.184:0/2793857525 >> v2:172.21.15.129:3300/0 conn(0x7f94480a8970 0x7f944813ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 -- 172.21.15.184:0/2793857525 >> 172.21.15.184:0/2793857525 conn(0x7f9448066dd0 msgr2=0x7f94480b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:21.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 -- 172.21.15.184:0/2793857525 shutdown_connections 2023-12-07T23:45:21.189 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.183+0000 7f944cdf0640 1 -- 172.21.15.184:0/2793857525 wait complete. 2023-12-07T23:45:21.192 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 10 monclient: build_initial_monmap 2023-12-07T23:45:21.192 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 10 monclient: monmap: 2023-12-07T23:45:21.192 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 0 2023-12-07T23:45:21.192 INFO:teuthology.orchestra.run.smithi184.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:21.192 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 0.000000 2023-12-07T23:45:21.192 INFO:teuthology.orchestra.run.smithi184.stderr:created 0.000000 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 librados: starting msgr at 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 librados: starting objecter 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 Processor -- start 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 -- start start 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 librados: setting wanted keys 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 librados: calling monclient init 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 10 monclient: init 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:21.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94480a8970 0x7f94480b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 10 monclient(hunting): picked mon.noname-a con 0x7f94480a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f94480b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.184+0000 7f944cdf0640 10 monclient(hunting): picked mon.noname-c con 0x7f94480a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94481421a0 0x7f94481dab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 10 monclient(hunting): picked mon.noname-b con 0x7f94481421a0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94480b6720 con 0x7f94480a8970 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94480ac2c0 con 0x7f94481421a0 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94480b7370 con 0x7f94480a9bd0 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:21.194 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944cdf0640 10 monclient(hunting): authenticate will time out at 1326.498338s 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f9445d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f94480b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94480a8970 0x7f94480b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f9445d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f94480b6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.184:44798/0 (socket says 172.21.15.184:44798) 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94480a8970 0x7f94480b6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.184:37288/0 (socket says 172.21.15.184:37288) 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f9445d9b640 1 -- 172.21.15.184:0/1789782570 learned_addr learned my addr 172.21.15.184:0/1789782570 (peer_addr_for_me v2:172.21.15.184:0/0) 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f9445d9b640 10 monclient(hunting): get_auth_request con 0x7f94480a9bd0 auth_method 0 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f9445d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f9445d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f9445d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944659c640 10 monclient(hunting): get_auth_request con 0x7f94480a8970 auth_method 0 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:21.195 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.185+0000 7f944659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.186+0000 7f9445d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.186+0000 7f9445d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.186+0000 7f9445d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.186+0000 7f944659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.186+0000 7f944659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.186+0000 7f944659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:21.197 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 1 -- 172.21.15.184:0/1789782570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94481421a0 msgr2=0x7f94481dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 1 --2- 172.21.15.184:0/1789782570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94481421a0 0x7f94481dab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 1 -- 172.21.15.184:0/1789782570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94480a8970 msgr2=0x7f94480b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 1 --2- 172.21.15.184:0/1789782570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94480a8970 0x7f94480b6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 10 monclient: _finish_hunting 0 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 1 monclient: found mon.noname-c 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:45:21.198 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.189+0000 7f9445d9b640 1 -- 172.21.15.184:0/1789782570 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94380094d0 con 0x7f94480a9bd0 2023-12-07T23:45:21.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.191+0000 7f9445d9b640 1 --2- 172.21.15.184:0/1789782570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f94480b6e30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94380090b0 tx=0x7f94380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:21.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.191+0000 7f9436ffd640 10 client.?.objecter ms_handle_connect 0x7f94480a9bd0 2023-12-07T23:45:21.199 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.191+0000 7f9436ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438033a10 con 0x7f94480a9bd0 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f9436ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f9436ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f9436ffd640 10 monclient: dump: 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:21.200 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f9436ffd640 10 monclient: _finish_auth 0 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f9436ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f9436ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 5 monclient: authenticate success, global_id 4142 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 10 monclient: _renew_subs 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94481db040 con 0x7f94480a9bd0 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 10 monclient: _renew_subs 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94481db540 con 0x7f94480a9bd0 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.193+0000 7f944cdf0640 1 librados: init done 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f944cdf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f944cdf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:21.201 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f944cdf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944804e840 con 0x7f94480a9bd0 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9438033b70 con 0x7f94480a9bd0 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f9436ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438051430 con 0x7f94480a9bd0 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f9436ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f9436ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f9436ffd640 10 monclient: dump: 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:epoch 1 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:election_strategy: 1 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:21.203 INFO:teuthology.orchestra.run.smithi184.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:21.203 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-07T23:45:21.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.194+0000 7f9444d99640 4 set_mon_vals no callback set 2023-12-07T23:45:21.226 INFO:tasks.ceph.mgr.y.smithi129.stderr:2023-12-07T23:45:21.224+0000 7f6055e4b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-07T23:45:21.294 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.291+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f943803a040 con 0x7f94480a9bd0 2023-12-07T23:45:21.294 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.291+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f943803ca60 con 0x7f94480a9bd0 2023-12-07T23:45:21.294 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.291+0000 7f9436ffd640 10 client.4142.objecter ms_dispatch 0x7f9448000b90 osd_map(9..9 src has 1..9) v4 2023-12-07T23:45:21.294 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.291+0000 7f9436ffd640 3 client.4142.objecter handle_osd_map got epochs [9,9] > 0 2023-12-07T23:45:21.294 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.291+0000 7f9436ffd640 3 client.4142.objecter handle_osd_map decoding full epoch 9 2023-12-07T23:45:21.295 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.291+0000 7f9436ffd640 20 client.4142.objecter dump_active .. 0 homeless 2023-12-07T23:45:21.300 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.298+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9438043020 con 0x7f94480a9bd0 2023-12-07T23:45:21.300 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.298+0000 7f9436ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:21.301 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.298+0000 7f9436ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:21.371 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.369+0000 7f944cdf0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "29e1d98a-ea92-42d8-b827-0004493a6525", "id": 7}] 2023-12-07T23:45:21.372 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.369+0000 7f944cdf0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "29e1d98a-ea92-42d8-b827-0004493a6525", "id": 7}] 2023-12-07T23:45:21.372 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.369+0000 7f944cdf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:21.372 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.369+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd new", "uuid": "29e1d98a-ea92-42d8-b827-0004493a6525", "id": 7} v 0) v1 -- 0x7f94480b2cf0 con 0x7f94480a9bd0 2023-12-07T23:45:21.536 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.534+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7f943803cc40 con 0x7f94480a9bd0 2023-12-07T23:45:21.939 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.936+0000 7f9436ffd640 1 -- 172.21.15.184:0/1789782570 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29e1d98a-ea92-42d8-b827-0004493a6525", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9438038020 con 0x7f94480a9bd0 2023-12-07T23:45:21.939 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.936+0000 7f9436ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "29e1d98a-ea92-42d8-b827-0004493a6525", "id": 7}] 2023-12-07T23:45:21.939 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.936+0000 7f9436ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:21.940 INFO:teuthology.orchestra.run.smithi184.stdout:7 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 10 librados: watch_flush enter 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 10 librados: watch_flush exit 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 20 client.4142.objecter shutdown clearing up homeless session... 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 10 client.4142.objecter successfully canceled tick 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 10 monclient: shutdown 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 msgr2=0x7f94480b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 1 --2- 172.21.15.184:0/1789782570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f94480b6e30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94380090b0 tx=0x7f94380046c0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 shutdown_connections 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 1 --2- 172.21.15.184:0/1789782570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94481421a0 0x7f94481dab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 1 --2- 172.21.15.184:0/1789782570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94480a9bd0 0x7f94480b6e30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 1 --2- 172.21.15.184:0/1789782570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94480a8970 0x7f94480b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:21.942 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.937+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 >> 172.21.15.184:0/1789782570 conn(0x7f9448066dd0 msgr2=0x7f9448145880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:21.942 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.938+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 shutdown_connections 2023-12-07T23:45:21.942 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.938+0000 7f944cdf0640 1 -- 172.21.15.184:0/1789782570 wait complete. 2023-12-07T23:45:21.942 INFO:teuthology.orchestra.run.smithi184.stderr:2023-12-07T23:45:21.938+0000 7f944cdf0640 1 librados: shutdown 2023-12-07T23:45:21.950 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.950 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-07T23:45:21.951 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-12-07T23:45:21.953 INFO:tasks.ceph.osd.0:Started 2023-12-07T23:45:21.953 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.953 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-07T23:45:21.953 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-12-07T23:45:21.955 INFO:tasks.ceph.osd.1:Started 2023-12-07T23:45:21.955 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.955 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-07T23:45:21.955 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-12-07T23:45:21.960 INFO:tasks.ceph.osd.2:Started 2023-12-07T23:45:21.960 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.961 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-07T23:45:21.961 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-12-07T23:45:21.963 INFO:tasks.ceph.osd.3:Started 2023-12-07T23:45:21.963 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.963 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-07T23:45:21.963 DEBUG:teuthology.orchestra.run.smithi184:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-12-07T23:45:21.965 INFO:tasks.ceph.osd.4:Started 2023-12-07T23:45:21.965 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.966 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-07T23:45:21.966 DEBUG:teuthology.orchestra.run.smithi184:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-07T23:45:21.967 INFO:tasks.ceph.osd.5:Started 2023-12-07T23:45:21.968 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.968 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-07T23:45:21.968 DEBUG:teuthology.orchestra.run.smithi184:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-12-07T23:45:21.970 INFO:tasks.ceph.osd.6:Started 2023-12-07T23:45:21.970 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T23:45:21.970 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-07T23:45:21.971 DEBUG:teuthology.orchestra.run.smithi184:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-07T23:45:21.973 INFO:tasks.ceph.osd.7:Started 2023-12-07T23:45:21.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.134+0000 7f57eaa3f640 10 monclient: get_monmap_and_config 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.134+0000 7f57eaa3f640 10 monclient: build_initial_monmap 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.134+0000 7f57eaa3f640 10 monclient: monmap: 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:45:22.135018+0000 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:45:22.135018+0000 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:22.137 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:22.138 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:22.138 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:22.138 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:22.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.135+0000 7f57eaa3f640 1 Processor -- start 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.136+0000 7f57eaa3f640 1 -- start start 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.136+0000 7f57eaa3f640 10 monclient: init 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.136+0000 7f57eaa3f640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57e40a8970 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f57e40a9bd0 0x7f57e41421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient(hunting): picked mon.noname-c con 0x7f57e40a9bd0 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e41426e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient(hunting): picked mon.noname-b con 0x7f57e413f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40aa950 con 0x7f57e40a8970 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e40a90e0 con 0x7f57e40a9bd0 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:22.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e40aabc0 con 0x7f57e413f580 2023-12-07T23:45:22.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:22.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.137+0000 7f57eaa3f640 10 monclient(hunting): authenticate will time out at 1322.324088s 2023-12-07T23:45:22.148 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.146+0000 7f57e8fdc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e41426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:22.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.158+0000 7f57e3fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f57e40a8970 0x7f57e413ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:54798/0 (socket says 172.21.15.129:54798) 2023-12-07T23:45:22.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.158+0000 7f57e3fff640 1 -- 172.21.15.129:0/1289847316 learned_addr learned my addr 172.21.15.129:0/1289847316 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:45:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.161+0000 7f57e8fdc640 10 monclient(hunting): get_auth_request con 0x7f57e413f580 auth_method 0 2023-12-07T23:45:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.161+0000 7f57e8fdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.161+0000 7f57e8fdc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.161+0000 7f57e8fdc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.169+0000 7f57e37fe640 1 --2- 172.21.15.129:0/1289847316 >> v2:172.21.15.184:3300/0 conn(0x7f57e40a9bd0 0x7f57e41421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.169+0000 7f57e37fe640 10 monclient(hunting): get_auth_request con 0x7f57e40a9bd0 auth_method 0 2023-12-07T23:45:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.169+0000 7f57e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.169+0000 7f57e37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.169+0000 7f57e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.170+0000 7f57e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.170+0000 7f57e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.170+0000 7f57e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:22.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 10 monclient(hunting): handle_auth_done global_id 4174 payload 274 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 1 -- 172.21.15.129:0/1289847316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 msgr2=0x7f57e41426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 1 --2- 172.21.15.129:0/1289847316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e41426e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 1 -- 172.21.15.129:0/1289847316 >> v1:172.21.15.129:6789/0 conn(0x7f57e40a8970 legacy=0x7f57e413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 10 monclient: _finish_hunting 0 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 1 monclient: found mon.noname-c 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:45:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.172+0000 7f57e37fe640 1 -- 172.21.15.129:0/1289847316 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40aaea0 con 0x7f57e40a9bd0 2023-12-07T23:45:22.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.175+0000 7f57e37fe640 1 --2- 172.21.15.129:0/1289847316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a9bd0 0x7f57e41421a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f57d80097a0 tx=0x7f57d802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee14eafa864de242 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 1 -- 172.21.15.129:0/1289847316 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d8030710 con 0x7f57e40a9bd0 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: dump: 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:22.185 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: _finish_auth 0 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 1 -- 172.21.15.129:0/1289847316 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d802e050 con 0x7f57e40a9bd0 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57eaa3f640 5 monclient: authenticate success, global_id 4174 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57eaa3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 1 -- 172.21.15.129:0/1289847316 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d8030a90 con 0x7f57e40a9bd0 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:22.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57e2ffd640 10 monclient: dump: 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57eaa3f640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.180+0000 7f57eaa3f640 4 set_mon_vals no callback set 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 10 monclient: shutdown 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:22.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 -- 172.21.15.129:0/1289847316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a9bd0 msgr2=0x7f57e41421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 --2- 172.21.15.129:0/1289847316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a9bd0 0x7f57e41421a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f57d80097a0 tx=0x7f57d802b710 comp rx=0 tx=0).stop 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 -- 172.21.15.129:0/1289847316 shutdown_connections 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 --2- 172.21.15.129:0/1289847316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e41426e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 --2- 172.21.15.129:0/1289847316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a9bd0 0x7f57e41421a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 -- 172.21.15.129:0/1289847316 >> 172.21.15.129:0/1289847316 conn(0x7f57e4066dd0 msgr2=0x7f57e40b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 -- 172.21.15.129:0/1289847316 shutdown_connections 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.181+0000 7f57eaa3f640 1 -- 172.21.15.129:0/1289847316 wait complete. 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.182+0000 7f57eaa3f640 10 monclient: build_initial_monmap 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.182+0000 7f57eaa3f640 10 monclient: monmap: 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:22.188 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.183+0000 7f57eaa3f640 1 librados: starting msgr at 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.183+0000 7f57eaa3f640 1 librados: starting objecter 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.183+0000 7f57eaa3f640 1 Processor -- start 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.183+0000 7f57eaa3f640 1 -- start start 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.183+0000 7f57eaa3f640 1 librados: setting wanted keys 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.183+0000 7f57eaa3f640 1 librados: calling monclient init 2023-12-07T23:45:22.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.183+0000 7f57eaa3f640 10 monclient: init 2023-12-07T23:45:22.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:22.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:45:22.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a8970 0x7f57e4137d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:22.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): picked mon.noname-b con 0x7f57e40a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e40a9bd0 0x7f57e413b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): picked mon.noname-a con 0x7f57e40a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e413a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): picked mon.noname-c con 0x7f57e413f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e41d6eb0 con 0x7f57e40a9bd0 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e41d7100 con 0x7f57e40a8970 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e413bb70 con 0x7f57e413f580 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.184+0000 7f57eaa3f640 10 monclient(hunting): authenticate will time out at 1322.371451s 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a8970 0x7f57e4137d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e8fdc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e413a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e8fdc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e413a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:56362/0 (socket says 172.21.15.129:56362) 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e8fdc640 1 -- 172.21.15.129:0/2598679261 learned_addr learned my addr 172.21.15.129:0/2598679261 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e8fdc640 10 monclient(hunting): get_auth_request con 0x7f57e413f580 auth_method 0 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e8fdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e8fdc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e8fdc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e3fff640 10 monclient(hunting): get_auth_request con 0x7f57e40a8970 auth_method 0 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.185+0000 7f57e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.186+0000 7f57e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.186+0000 7f57e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.186+0000 7f57e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.186+0000 7f57e8fdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.186+0000 7f57e8fdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.186+0000 7f57e8fdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 10 monclient(hunting): handle_auth_done global_id 4177 payload 824 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 1 -- 172.21.15.129:0/2598679261 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 msgr2=0x7f57e413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 1 --2- 172.21.15.129:0/2598679261 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e413a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 1 -- 172.21.15.129:0/2598679261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e40a9bd0 msgr2=0x7f57e413b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 1 --2- 172.21.15.129:0/2598679261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e40a9bd0 0x7f57e413b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 10 monclient: _finish_hunting 0 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 1 monclient: found mon.noname-b 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:22.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.190+0000 7f57e3fff640 1 -- 172.21.15.129:0/2598679261 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d8009400 con 0x7f57e40a8970 2023-12-07T23:45:22.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.197+0000 7f57e3fff640 1 --2- 172.21.15.129:0/2598679261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a8970 0x7f57e4137d30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f57d400c410 tx=0x7f57d400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:22.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.197+0000 7f57e0ff9640 10 client.?.objecter ms_handle_connect 0x7f57e40a8970 2023-12-07T23:45:22.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.197+0000 7f57e0ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 1 -- 172.21.15.129:0/2598679261 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4005ce0 con 0x7f57e40a8970 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: dump: 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: _finish_auth 0 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: _check_auth_tickets 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 1 -- 172.21.15.129:0/2598679261 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d4005e40 con 0x7f57e40a8970 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 1 -- 172.21.15.129:0/2598679261 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4007310 con 0x7f57e40a8970 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e0ff9640 10 monclient: dump: 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57e27fc640 4 set_mon_vals no callback set 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 5 monclient: authenticate success, global_id 4177 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 10 monclient: _renew_subs 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e413ae30 con 0x7f57e40a8970 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 10 monclient: _renew_subs 2023-12-07T23:45:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e41d7800 con 0x7f57e40a8970 2023-12-07T23:45:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.202+0000 7f57eaa3f640 1 librados: init done 2023-12-07T23:45:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.204+0000 7f57eaa3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.204+0000 7f57eaa3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.204+0000 7f57eaa3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.204+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e404e840 con 0x7f57e40a8970 2023-12-07T23:45:22.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.222+0000 7f57e0ff9640 1 -- 172.21.15.129:0/2598679261 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7f57d4007470 con 0x7f57e40a8970 2023-12-07T23:45:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.405+0000 7f57e0ff9640 1 -- 172.21.15.129:0/2598679261 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f57d40109d0 con 0x7f57e40a8970 2023-12-07T23:45:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.406+0000 7f57e0ff9640 10 client.4177.objecter ms_dispatch 0x7f57e4000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T23:45:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.406+0000 7f57e0ff9640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T23:45:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.406+0000 7f57e0ff9640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2023-12-07T23:45:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.406+0000 7f57e0ff9640 20 client.4177.objecter dump_active .. 0 homeless 2023-12-07T23:45:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.406+0000 7f57e0ff9640 1 -- 172.21.15.129:0/2598679261 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f57d4007720 con 0x7f57e40a8970 2023-12-07T23:45:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.406+0000 7f57e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.406+0000 7f57e0ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:22.528 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.525+0000 7f57eaa3f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:22.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.525+0000 7f57eaa3f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:22.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.525+0000 7f57eaa3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:22.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.525+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57e41d72c0 con 0x7f57e40a8970 2023-12-07T23:45:22.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.599+0000 7f57e0ff9640 1 -- 172.21.15.129:0/2598679261 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f57d4017030 con 0x7f57e40a8970 2023-12-07T23:45:22.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.599+0000 7f57e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:22.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.599+0000 7f57e0ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:22.602 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:45:22.602 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":10,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:45:21.546459+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:45:22.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 10 librados: watch_flush enter 2023-12-07T23:45:22.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 10 librados: watch_flush exit 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 20 client.4177.objecter shutdown clearing up homeless session... 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 10 client.4177.objecter successfully canceled tick 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 10 monclient: shutdown 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a8970 msgr2=0x7f57e4137d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 --2- 172.21.15.129:0/2598679261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a8970 0x7f57e4137d30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f57d400c410 tx=0x7f57d400c8d0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 shutdown_connections 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 --2- 172.21.15.129:0/2598679261 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e413f580 0x7f57e413a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 --2- 172.21.15.129:0/2598679261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e40a9bd0 0x7f57e413b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 --2- 172.21.15.129:0/2598679261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40a8970 0x7f57e4137d30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 >> 172.21.15.129:0/2598679261 conn(0x7f57e4066dd0 msgr2=0x7f57e40b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 shutdown_connections 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 -- 172.21.15.129:0/2598679261 wait complete. 2023-12-07T23:45:22.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:45:22.600+0000 7f57eaa3f640 1 librados: shutdown 2023-12-07T23:45:22.621 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-07T23:45:22.621 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-07T23:45:23.205 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:23.202+0000 7f5777851640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-07T23:45:23.250 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-07T23:45:23.246+0000 7f5777851640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-07T23:45:23.428 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.571+0000 7fbf794a9640 10 monclient: get_monmap_and_config 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.571+0000 7fbf794a9640 10 monclient: build_initial_monmap 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.571+0000 7fbf794a9640 10 monclient: monmap: 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:45:23.572906+0000 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:45:23.572906+0000 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:23.575 INFO:teuthology.misc.health.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:23.576 INFO:teuthology.misc.health.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:23.576 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:23.576 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.573+0000 7fbf794a9640 1 Processor -- start 2023-12-07T23:45:23.576 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.573+0000 7fbf794a9640 1 -- start start 2023-12-07T23:45:23.576 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.573+0000 7fbf794a9640 10 monclient: init 2023-12-07T23:45:23.578 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.574+0000 7fbf794a9640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 0x7fbf741459b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient(hunting): picked mon.noname-b con 0x7fbf740a8990 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbf740a8d70 0x7fbf7413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient(hunting): picked mon.noname-c con 0x7fbf740a8d70 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf74145ef0 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74145520 con 0x7fbf74145ef0 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf740aabe0 con 0x7fbf740a8d70 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740aaec0 con 0x7fbf740a8990 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:23.579 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.575+0000 7fbf794a9640 10 monclient(hunting): authenticate will time out at 1323.763256s 2023-12-07T23:45:23.580 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.577+0000 7fbf6bfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbf740a8d70 0x7fbf7413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:23.581 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.577+0000 7fbf6bfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbf740a8d70 0x7fbf7413f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:47032/0 (socket says 172.21.15.129:47032) 2023-12-07T23:45:23.581 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.577+0000 7fbf6bfff640 1 -- 172.21.15.129:0/3066242831 learned_addr learned my addr 172.21.15.129:0/3066242831 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:23.581 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.577+0000 7fbf6bfff640 10 monclient(hunting): get_auth_request con 0x7fbf740a8d70 auth_method 0 2023-12-07T23:45:23.581 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.577+0000 7fbf6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:23.581 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.577+0000 7fbf6bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:23.581 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.577+0000 7fbf6bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:23.582 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.578+0000 7fbf6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:23.582 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.578+0000 7fbf6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:23.582 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.578+0000 7fbf6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:23.585 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.581+0000 7fbf6bfff640 10 monclient(hunting): handle_auth_done global_id 4180 payload 274 2023-12-07T23:45:23.585 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.581+0000 7fbf6bfff640 1 -- 172.21.15.129:0/3066242831 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 msgr2=0x7fbf741459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:23.585 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.582+0000 7fbf6bfff640 1 --2- 172.21.15.129:0/3066242831 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 0x7fbf741459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:23.585 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.582+0000 7fbf6bfff640 1 -- 172.21.15.129:0/3066242831 >> v1:172.21.15.184:6789/0 conn(0x7fbf74145ef0 legacy=0x7fbf74141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:23.585 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.582+0000 7fbf6bfff640 10 monclient: _finish_hunting 0 2023-12-07T23:45:23.585 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.582+0000 7fbf6bfff640 1 monclient: found mon.noname-c 2023-12-07T23:45:23.585 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.582+0000 7fbf6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:23.586 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.582+0000 7fbf6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:45:23.586 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.582+0000 7fbf6bfff640 1 -- 172.21.15.129:0/3066242831 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740a97c0 con 0x7fbf740a8d70 2023-12-07T23:45:23.592 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.589+0000 7fbf6bfff640 1 --2- 172.21.15.129:0/3066242831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf740a8d70 0x7fbf7413f080 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009870 tx=0x7fbf5c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae4307faacb066e server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.594+0000 7fbf727fc640 1 -- 172.21.15.129:0/3066242831 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c030840 con 0x7fbf740a8d70 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.594+0000 7fbf727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: dump: 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:23.600 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: _finish_auth 0 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: _check_auth_tickets 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 1 -- 172.21.15.129:0/3066242831 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c0309a0 con 0x7fbf740a8d70 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf794a9640 5 monclient: authenticate success, global_id 4180 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf794a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 1 -- 172.21.15.129:0/3066242831 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c030d30 con 0x7fbf740a8d70 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf727fc640 10 monclient: dump: 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:23.601 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf794a9640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.595+0000 7fbf794a9640 4 set_mon_vals no callback set 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 10 monclient: shutdown 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 1 -- 172.21.15.129:0/3066242831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf740a8d70 msgr2=0x7fbf7413f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:23.602 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 1 --2- 172.21.15.129:0/3066242831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf740a8d70 0x7fbf7413f080 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009870 tx=0x7fbf5c02b980 comp rx=0 tx=0).stop 2023-12-07T23:45:23.603 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 1 -- 172.21.15.129:0/3066242831 shutdown_connections 2023-12-07T23:45:23.603 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 1 --2- 172.21.15.129:0/3066242831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf740a8d70 0x7fbf7413f080 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:23.603 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 1 --2- 172.21.15.129:0/3066242831 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 0x7fbf741459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:23.603 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 1 -- 172.21.15.129:0/3066242831 >> 172.21.15.129:0/3066242831 conn(0x7fbf74066dd0 msgr2=0x7fbf740b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:23.603 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.596+0000 7fbf794a9640 1 -- 172.21.15.129:0/3066242831 shutdown_connections 2023-12-07T23:45:23.603 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.597+0000 7fbf794a9640 1 -- 172.21.15.129:0/3066242831 wait complete. 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.598+0000 7fbf794a9640 10 monclient: build_initial_monmap 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.598+0000 7fbf794a9640 10 monclient: monmap: 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:last_changed 0.000000 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:created 0.000000 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:23.608 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.599+0000 7fbf794a9640 1 librados: starting msgr at 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.599+0000 7fbf794a9640 1 librados: starting objecter 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.599+0000 7fbf794a9640 1 Processor -- start 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.599+0000 7fbf794a9640 1 -- start start 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.599+0000 7fbf794a9640 1 librados: setting wanted keys 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.599+0000 7fbf794a9640 1 librados: calling monclient init 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.599+0000 7fbf794a9640 10 monclient: init 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.600+0000 7fbf794a9640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 0x7fbf740b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): picked mon.noname-c con 0x7fbf740a8990 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf740a8d70 0x7fbf74141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): picked mon.noname-a con 0x7fbf740a8d70 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf74145ef0 0x7fbf74135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): picked mon.noname-b con 0x7fbf74145ef0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:23.609 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf741425e0 con 0x7fbf740a8d70 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74141ea0 con 0x7fbf74145ef0 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74142150 con 0x7fbf740a8990 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.601+0000 7fbf794a9640 10 monclient(hunting): authenticate will time out at 1323.789895s 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf737fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf74145ef0 0x7fbf74135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf737fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf74145ef0 0x7fbf74135d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:47040/0 (socket says 172.21.15.129:47040) 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf737fe640 1 -- 172.21.15.129:0/1481576018 learned_addr learned my addr 172.21.15.129:0/1481576018 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf737fe640 10 monclient(hunting): get_auth_request con 0x7fbf74145ef0 auth_method 0 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf737fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:23.610 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf737fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf72ffd640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 0x7fbf740b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf72ffd640 10 monclient(hunting): get_auth_request con 0x7fbf740a8990 auth_method 0 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf72ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.602+0000 7fbf72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf6bfff640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf740a8d70 0x7fbf74141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf6bfff640 10 monclient(hunting): get_auth_request con 0x7fbf740a8d70 auth_method 0 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf6bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:23.611 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.603+0000 7fbf6bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 10 monclient(hunting): handle_auth_done global_id 4183 payload 824 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 1 -- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 msgr2=0x7fbf740b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 0x7fbf740b6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 1 -- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf740a8d70 msgr2=0x7fbf74141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf740a8d70 0x7fbf74141960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 10 monclient: _finish_hunting 0 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 1 monclient: found mon.noname-b 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:23.612 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:23.613 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.608+0000 7fbf737fe640 1 -- 172.21.15.129:0/1481576018 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0094d0 con 0x7fbf74145ef0 2023-12-07T23:45:23.615 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.612+0000 7fbf737fe640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf74145ef0 0x7fbf74135d40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00c760 tx=0x7fbf6c00cb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:23.615 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.613+0000 7fbf6b7fe640 10 client.?.objecter ms_handle_connect 0x7fbf74145ef0 2023-12-07T23:45:23.616 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.613+0000 7fbf6b7fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c006d50 con 0x7fbf74145ef0 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf6b7fe640 10 monclient: dump: 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:23.619 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf6b7fe640 10 monclient: _finish_auth 0 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf6b7fe640 10 monclient: _check_auth_tickets 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 5 monclient: authenticate success, global_id 4183 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 10 monclient: _renew_subs 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf74140ff0 con 0x7fbf74145ef0 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 10 monclient: _renew_subs 2023-12-07T23:45:23.620 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:23.621 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf74136460 con 0x7fbf74145ef0 2023-12-07T23:45:23.621 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.616+0000 7fbf794a9640 1 librados: init done 2023-12-07T23:45:23.621 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.617+0000 7fbf794a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:23.621 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.617+0000 7fbf794a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:23.621 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.617+0000 7fbf794a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:23.621 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.617+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c003970 con 0x7fbf74145ef0 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.617+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf740ab840 con 0x7fbf74145ef0 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.618+0000 7fbf6b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.618+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003ce0 con 0x7fbf74145ef0 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.618+0000 7fbf6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.618+0000 7fbf6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.618+0000 7fbf6b7fe640 10 monclient: dump: 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:23.622 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:23.623 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:23.623 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:23.623 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:23.618+0000 7fbf71ffb640 4 set_mon_vals no callback set 2023-12-07T23:45:24.060 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.057+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 3) v1 ==== 89069+0+0 (secure 0 0 0) 0x7fbf6c0050d0 con 0x7fbf74145ef0 2023-12-07T23:45:24.061 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.058+0000 7fbf6b7fe640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fbf5806c670 0x7fbf5806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:24.061 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.058+0000 7fbf72ffd640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fbf5806c670 0x7fbf5806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:24.061 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.058+0000 7fbf72ffd640 10 monclient: get_auth_request con 0x7fbf5806c670 auth_method 0 2023-12-07T23:45:24.061 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.058+0000 7fbf6b7fe640 10 client.4183.objecter ms_handle_connect 0x7fbf5806c670 2023-12-07T23:45:24.061 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.058+0000 7fbf72ffd640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fbf5806c670 0x7fbf5806eb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009890 tx=0x7fbf64008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:24.086 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.083+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbf6c059310 con 0x7fbf74145ef0 2023-12-07T23:45:24.086 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.083+0000 7fbf6b7fe640 10 client.4183.objecter ms_dispatch 0x7fbf74000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T23:45:24.086 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.083+0000 7fbf6b7fe640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T23:45:24.086 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.083+0000 7fbf6b7fe640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2023-12-07T23:45:24.086 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.083+0000 7fbf6b7fe640 20 client.4183.objecter dump_active .. 0 homeless 2023-12-07T23:45:24.087 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.085+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf6c054f50 con 0x7fbf74145ef0 2023-12-07T23:45:24.087 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.085+0000 7fbf6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:24.087 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.085+0000 7fbf6b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:24.445 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.441+0000 7fbf794a9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:24.445 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.441+0000 7fbf794a9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:24.445 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.441+0000 7fbf794a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:24.445 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.441+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf74075c40 con 0x7fbf74145ef0 2023-12-07T23:45:24.601 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.598+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fbf6c055320 con 0x7fbf74145ef0 2023-12-07T23:45:24.747 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.744+0000 7fbf6b7fe640 1 -- 172.21.15.129:0/1481576018 <== mon.1 v2:172.21.15.184:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbf6c03b650 con 0x7fbf74145ef0 2023-12-07T23:45:24.747 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.744+0000 7fbf6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:24.747 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.744+0000 7fbf6b7fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:24.748 INFO:teuthology.misc.health.smithi129.stdout: 2023-12-07T23:45:24.748 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:45:21.546459+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 10 librados: watch_flush enter 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 10 librados: watch_flush exit 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 20 client.4183.objecter shutdown clearing up homeless session... 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 10 client.4183.objecter successfully canceled tick 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fbf5806c670 msgr2=0x7fbf5806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fbf5806c670 0x7fbf5806eb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009890 tx=0x7fbf64008040 comp rx=0 tx=0).stop 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 10 monclient: shutdown 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:24.751 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf74145ef0 msgr2=0x7fbf74135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:24.752 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.748+0000 7fbf794a9640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf74145ef0 0x7fbf74135d40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00c760 tx=0x7fbf6c00cb60 comp rx=0 tx=0).stop 2023-12-07T23:45:24.752 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 shutdown_connections 2023-12-07T23:45:24.752 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fbf5806c670 0x7fbf5806eb30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:24.752 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf74145ef0 0x7fbf74135d40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:24.752 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf740a8d70 0x7fbf74141960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:24.752 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 --2- 172.21.15.129:0/1481576018 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf740a8990 0x7fbf740b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:24.753 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 >> 172.21.15.129:0/1481576018 conn(0x7fbf74066dd0 msgr2=0x7fbf740b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:24.753 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 shutdown_connections 2023-12-07T23:45:24.753 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 -- 172.21.15.129:0/1481576018 wait complete. 2023-12-07T23:45:24.753 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:24.749+0000 7fbf794a9640 1 librados: shutdown 2023-12-07T23:45:24.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T23:45:29.967 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:07.931 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:29.975 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:07.940 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.028 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:07.993 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.037 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:08.003 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.261 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:08.230 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.324 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:08.293 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.453 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:08.413 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.473 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:08.440 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.516 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:08.487 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.517 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:08.476 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.529 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:08.487 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.537 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:08.504 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.572 INFO:tasks.ceph.osd.7.smithi184.stderr:2023-12-07T23:45:30.564+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:30.575 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:08.547 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.585 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:45:30.577+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:30.590 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:08.548 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.626 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:08.601 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.684 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:08.659 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:30.890 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:45:30.881+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:31.014 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:08.980 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.015 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:08.979 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.062 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:45:31.051+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:31.086 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:45:31.077+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:31.147 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:45:31.138+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:31.156 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:45:31.147+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:31.268 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:45:31.257+0000 5dc2340 -1 Falling back to public interface 2023-12-07T23:45:31.381 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:09.347 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.394 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:09.364 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.403 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:09.368 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.489 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:09.448 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.591 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T23:45:31.592 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:09.559 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.600 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:09.571 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.644 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:09.601 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.677+0000 7fe719d7d640 10 monclient: get_monmap_and_config 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.677+0000 7fe719d7d640 10 monclient: build_initial_monmap 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.677+0000 7fe719d7d640 10 monclient: monmap: 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:45:31.678277+0000 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:45:31.678277+0000 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:31.680 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:31.681 INFO:teuthology.misc.health.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:31.681 INFO:teuthology.misc.health.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:31.681 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:31.681 INFO:teuthology.misc.health.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:31.681 INFO:teuthology.misc.health.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:31.681 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.677+0000 7fe719d7d640 1 Processor -- start 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.677+0000 7fe719d7d640 1 -- start start 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.677+0000 7fe719d7d640 10 monclient: init 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe7140a8930 0x7fe7140a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7140a8930 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7140b21a0 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe71413f120 0x7fe71413f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe71413f120 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7140b1c20 con 0x7fe7140b21a0 2023-12-07T23:45:31.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140b25e0 con 0x7fe71413f120 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140b1e70 con 0x7fe7140a8930 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.678+0000 7fe719d7d640 10 monclient(hunting): authenticate will time out at 1331.866577s 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe7137fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe7140a8930 0x7fe7140a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe7137fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe7140a8930 0x7fe7140a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:57340/0 (socket says 172.21.15.129:57340) 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe7137fe640 1 -- 172.21.15.129:0/1060648166 learned_addr learned my addr 172.21.15.129:0/1060648166 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe7137fe640 10 monclient(hunting): get_auth_request con 0x7fe7140a8930 auth_method 0 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe7137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:31.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe7137fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe7137fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.679+0000 7fe713fff640 1 --2- 172.21.15.129:0/1060648166 >> v2:172.21.15.129:3300/0 conn(0x7fe71413f120 0x7fe71413f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.680+0000 7fe7137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.680+0000 7fe7137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.680+0000 7fe7137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.680+0000 7fe713fff640 10 monclient(hunting): get_auth_request con 0x7fe71413f120 auth_method 0 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.680+0000 7fe713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.680+0000 7fe713fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:31.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.680+0000 7fe713fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 10 monclient(hunting): handle_auth_done global_id 4166 payload 274 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 1 -- 172.21.15.129:0/1060648166 >> v2:172.21.15.129:3300/0 conn(0x7fe71413f120 msgr2=0x7fe71413f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 1 --2- 172.21.15.129:0/1060648166 >> v2:172.21.15.129:3300/0 conn(0x7fe71413f120 0x7fe71413f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 1 -- 172.21.15.129:0/1060648166 >> v1:172.21.15.129:6789/0 conn(0x7fe7140b21a0 legacy=0x7fe714142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 10 monclient: _finish_hunting 0 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 1 monclient: found mon.noname-b 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.681+0000 7fe7137fe640 1 -- 172.21.15.129:0/1060648166 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140aa9a0 con 0x7fe7140a8930 2023-12-07T23:45:31.687 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.683+0000 7fe7137fe640 1 --2- 172.21.15.129:0/1060648166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe7140a8930 0x7fe7140a8d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe704009870 tx=0x7fe70402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c68c124f91df55a1 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:31.691 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 1 -- 172.21.15.129:0/1060648166 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704031640 con 0x7fe7140a8930 2023-12-07T23:45:31.691 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:31.691 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: dump: 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:31.692 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:31.693 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: _finish_auth 0 2023-12-07T23:45:31.693 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:45:31.693 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:31.693 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 1 -- 172.21.15.129:0/1060648166 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7040317a0 con 0x7fe7140a8930 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe719d7d640 5 monclient: authenticate success, global_id 4166 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe719d7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe719d7d640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 1 -- 172.21.15.129:0/1060648166 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704031b30 con 0x7fe7140a8930 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe719d7d640 4 set_mon_vals no callback set 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.687+0000 7fe712ffd640 10 monclient: dump: 2023-12-07T23:45:31.694 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 10 monclient: shutdown 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 -- 172.21.15.129:0/1060648166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe7140a8930 msgr2=0x7fe7140a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 --2- 172.21.15.129:0/1060648166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe7140a8930 0x7fe7140a8d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe704009870 tx=0x7fe70402b8f0 comp rx=0 tx=0).stop 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 -- 172.21.15.129:0/1060648166 shutdown_connections 2023-12-07T23:45:31.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 --2- 172.21.15.129:0/1060648166 >> v2:172.21.15.129:3300/0 conn(0x7fe71413f120 0x7fe71413f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:31.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 --2- 172.21.15.129:0/1060648166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe7140a8930 0x7fe7140a8d10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:31.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 -- 172.21.15.129:0/1060648166 >> 172.21.15.129:0/1060648166 conn(0x7fe7140b77a0 msgr2=0x7fe7140b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:31.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 -- 172.21.15.129:0/1060648166 shutdown_connections 2023-12-07T23:45:31.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.688+0000 7fe719d7d640 1 -- 172.21.15.129:0/1060648166 wait complete. 2023-12-07T23:45:31.697 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:09.662 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.689+0000 7fe719d7d640 10 monclient: build_initial_monmap 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.689+0000 7fe719d7d640 10 monclient: monmap: 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:last_changed 0.000000 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:created 0.000000 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:31.700 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.689+0000 7fe719d7d640 1 librados: starting msgr at 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.689+0000 7fe719d7d640 1 librados: starting objecter 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.689+0000 7fe719d7d640 1 Processor -- start 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 -- start start 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 librados: setting wanted keys 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 librados: calling monclient init 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient: init 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7140a8930 0x7fe714139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7140a8930 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7140b21a0 0x7fe714135e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7140b21a0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:31.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe71413f120 0x7fe714139ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): picked mon.noname-c con 0x7fe71413f120 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71413b660 con 0x7fe7140a8930 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71413b8d0 con 0x7fe7140b21a0 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe714136370 con 0x7fe71413f120 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.690+0000 7fe719d7d640 10 monclient(hunting): authenticate will time out at 1331.878649s 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe713fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe71413f120 0x7fe714139ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe7137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7140a8930 0x7fe714139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:31.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe70affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7140b21a0 0x7fe714135e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe713fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe71413f120 0x7fe714139ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:57342/0 (socket says 172.21.15.129:57342) 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe713fff640 1 -- 172.21.15.129:0/730423155 learned_addr learned my addr 172.21.15.129:0/730423155 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe713fff640 10 monclient(hunting): get_auth_request con 0x7fe71413f120 auth_method 0 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe713fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe713fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe70affd640 10 monclient(hunting): get_auth_request con 0x7fe7140b21a0 auth_method 0 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe70affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe70affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe70affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe7137fe640 10 monclient(hunting): get_auth_request con 0x7fe7140a8930 auth_method 0 2023-12-07T23:45:31.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe7137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe7137fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.691+0000 7fe7137fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe70affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe70affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe70affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe7137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe7137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.692+0000 7fe7137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:31.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.694+0000 7fe70affd640 10 monclient(hunting): handle_auth_done global_id 4213 payload 824 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 1 -- 172.21.15.129:0/730423155 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe71413f120 msgr2=0x7fe714139ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe71413f120 0x7fe714139ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 1 -- 172.21.15.129:0/730423155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7140a8930 msgr2=0x7fe714139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7140a8930 0x7fe714139470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 10 monclient: _finish_hunting 0 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 1 monclient: found mon.noname-b 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:45:31.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.695+0000 7fe70affd640 1 -- 172.21.15.129:0/730423155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040094d0 con 0x7fe7140b21a0 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.697+0000 7fe70affd640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7140b21a0 0x7fe714135e30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c340 tx=0x7fe6fc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.697+0000 7fe710ff9640 10 client.?.objecter ms_handle_connect 0x7fe7140b21a0 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.697+0000 7fe710ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 1 -- 172.21.15.129:0/730423155 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc0052d0 con 0x7fe7140b21a0 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: dump: 2023-12-07T23:45:31.707 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: _finish_auth 0 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: _check_auth_tickets 2023-12-07T23:45:31.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 5 monclient: authenticate success, global_id 4213 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 1 -- 172.21.15.129:0/730423155 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc005450 con 0x7fe7140b21a0 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 1 -- 172.21.15.129:0/730423155 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc007260 con 0x7fe7140b21a0 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe710ff9640 10 monclient: dump: 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:31.709 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe7127fc640 4 set_mon_vals no callback set 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 10 monclient: _renew_subs 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe714137df0 con 0x7fe7140b21a0 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 10 monclient: _renew_subs 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:31.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71413a600 con 0x7fe7140b21a0 2023-12-07T23:45:31.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.701+0000 7fe719d7d640 1 librados: init done 2023-12-07T23:45:31.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.702+0000 7fe719d7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:31.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.702+0000 7fe719d7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:31.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.702+0000 7fe719d7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:31.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.702+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe714067b90 con 0x7fe7140b21a0 2023-12-07T23:45:31.749 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:09.714 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.770 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:09.745 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.792 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:09.762 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.842 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:09.802 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.949 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:09.915 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.950 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.947+0000 7fe710ff9640 1 -- 172.21.15.129:0/730423155 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fe6fc0073c0 con 0x7fe7140b21a0 2023-12-07T23:45:31.951 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.947+0000 7fe710ff9640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe6f406c7e0 0x7fe6f406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:31.951 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.948+0000 7fe7137fe640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe6f406c7e0 0x7fe6f406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:31.951 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.948+0000 7fe7137fe640 10 monclient: get_auth_request con 0x7fe6f406c7e0 auth_method 0 2023-12-07T23:45:31.951 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.948+0000 7fe710ff9640 10 client.4213.objecter ms_handle_connect 0x7fe6f406c7e0 2023-12-07T23:45:31.951 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.948+0000 7fe7137fe640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe6f406c7e0 0x7fe6f406eca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe704036000 tx=0x7fe70403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:31.952 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:09.924 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.956 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.953+0000 7fe710ff9640 1 -- 172.21.15.129:0/730423155 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe6fc057c10 con 0x7fe7140b21a0 2023-12-07T23:45:31.956 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.953+0000 7fe710ff9640 10 client.4213.objecter ms_dispatch 0x7fe714000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T23:45:31.956 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.953+0000 7fe710ff9640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T23:45:31.957 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.953+0000 7fe710ff9640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2023-12-07T23:45:31.957 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.953+0000 7fe710ff9640 20 client.4213.objecter dump_active .. 0 homeless 2023-12-07T23:45:31.966 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:09.932 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:31.966 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.963+0000 7fe710ff9640 1 -- 172.21.15.129:0/730423155 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6fc053f50 con 0x7fe7140b21a0 2023-12-07T23:45:31.966 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.964+0000 7fe710ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:31.967 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:31.964+0000 7fe710ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:32.001 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:09.958 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.017 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:09.982 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.113 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:10.083 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.128 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:10.103 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.143 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.140+0000 7fe719d7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:32.143 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.140+0000 7fe719d7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:32.143 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.140+0000 7fe719d7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:45:32.143 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.140+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe714139c00 con 0x7fe7140b21a0 2023-12-07T23:45:32.157 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.154+0000 7fe710ff9640 1 -- 172.21.15.129:0/730423155 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe714139c00 con 0x7fe7140b21a0 2023-12-07T23:45:32.157 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.154+0000 7fe710ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:32.157 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.154+0000 7fe710ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:32.157 INFO:teuthology.misc.health.smithi129.stdout: 2023-12-07T23:45:32.158 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:45:21.546459+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 10 librados: watch_flush enter 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 10 librados: watch_flush exit 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 20 client.4213.objecter shutdown clearing up homeless session... 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 10 client.4213.objecter successfully canceled tick 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe6f406c7e0 msgr2=0x7fe6f406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe6f406c7e0 0x7fe6f406eca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe704036000 tx=0x7fe70403c040 comp rx=0 tx=0).stop 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 10 monclient: shutdown 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:32.160 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7140b21a0 msgr2=0x7fe714135e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7140b21a0 0x7fe714135e30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c340 tx=0x7fe6fc00c800 comp rx=0 tx=0).stop 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 shutdown_connections 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe6f406c7e0 0x7fe6f406eca0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe71413f120 0x7fe714139ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7140b21a0 0x7fe714135e30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 --2- 172.21.15.129:0/730423155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7140a8930 0x7fe714139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 >> 172.21.15.129:0/730423155 conn(0x7fe7140b77a0 msgr2=0x7fe71413dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 shutdown_connections 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 -- 172.21.15.129:0/730423155 wait complete. 2023-12-07T23:45:32.161 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:32.156+0000 7fe719d7d640 1 librados: shutdown 2023-12-07T23:45:32.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T23:45:32.170 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:10.129 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.205 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:10.180 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.237 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:10.203 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.265 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:10.232 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.275 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:10.246 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.283 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:10.248 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.314 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:10.272 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.382 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:10.351 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.434 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:10.394 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.474 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:10.449 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.504 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:10.470 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.530 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:10.497 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.540 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:10.511 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.556 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:10.520 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.582 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:10.540 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.645 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:10.614 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.700 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:10.660 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.742 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:10.717 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.768 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:10.734 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.798 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:10.765 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.805 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:10.776 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.827 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:10.792 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.846 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:10.804 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.909 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:10.879 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:32.968 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:10.928 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.006 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:10.981 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.032 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:10.998 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.068 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:11.039 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.069 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:11.036 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.092 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:11.057 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.110 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:11.068 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.174 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:11.144 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.245 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:11.205 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.272 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:11.247 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.300 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:11.266 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.337 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:11.308 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.344 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:11.311 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.364 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:11.329 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.379 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:11.337 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.442 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:11.412 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.510 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:11.470 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.542 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:11.517 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.611 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:11.578 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.617 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:11.588 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.647 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:11.605 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.708 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:11.677 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.774 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:11.734 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.775 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:11.741 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.807 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:11.782 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.829 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:11.795 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.839 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:11.804 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.876 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:11.843 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.891 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:11.862 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.894 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:11.859 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:33.912 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:11.870 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.182 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:12.151 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.236 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:12.206 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.263 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:12.223 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.288 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:12.263 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.321 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:12.280 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.343 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:12.318 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.355 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:12.321 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.395 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:12.367 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.409 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:12.376 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.453 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:12.424 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.465 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:12.422 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:34.519 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:12.477 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:38.975 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.088+0000 7f608fcbb640 10 monclient: get_monmap_and_config 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.088+0000 7f608fcbb640 10 monclient: build_initial_monmap 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.088+0000 7f608fcbb640 10 monclient: monmap: 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:45:39.089926+0000 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:45:39.089926+0000 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:39.092 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.089+0000 7f608fcbb640 1 Processor -- start 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.089+0000 7f608fcbb640 1 -- start start 2023-12-07T23:45:39.093 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.089+0000 7f608fcbb640 10 monclient: init 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f60880a8930 0x7f60880a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient(hunting): picked mon.noname-b con 0x7f60880a8930 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60880b21a0 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f608813f120 0x7f608813f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient(hunting): picked mon.noname-c con 0x7f608813f120 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60880b1c20 con 0x7f60880b21a0 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60880b25e0 con 0x7f608813f120 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:39.096 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60880b1e70 con 0x7f60880a8930 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.090+0000 7f608fcbb640 10 monclient(hunting): authenticate will time out at 1339.278795s 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608e258640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f608813f120 0x7f608813f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608da57640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f60880a8930 0x7f60880a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608da57640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f60880a8930 0x7f60880a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:58570/0 (socket says 172.21.15.129:58570) 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608da57640 1 -- 172.21.15.129:0/2737131272 learned_addr learned my addr 172.21.15.129:0/2737131272 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608da57640 10 monclient(hunting): get_auth_request con 0x7f60880a8930 auth_method 0 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608da57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:39.097 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608da57640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608da57640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608e258640 10 monclient(hunting): get_auth_request con 0x7f608813f120 auth_method 0 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608e258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608e258640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.091+0000 7f608e258640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.092+0000 7f608da57640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.092+0000 7f608da57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:39.098 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.092+0000 7f608da57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:39.099 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.092+0000 7f608e258640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:39.100 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.092+0000 7f608e258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:39.100 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.093+0000 7f608e258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:39.101 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2023-12-07T23:45:39.101 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 1 -- 172.21.15.129:0/2737131272 >> v2:172.21.15.184:3300/0 conn(0x7f608813f120 msgr2=0x7f608813f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:39.101 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 1 --2- 172.21.15.129:0/2737131272 >> v2:172.21.15.184:3300/0 conn(0x7f608813f120 0x7f608813f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.101 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 1 -- 172.21.15.129:0/2737131272 >> v1:172.21.15.184:6789/0 conn(0x7f60880b21a0 legacy=0x7f6088142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:39.101 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 10 monclient: _finish_hunting 0 2023-12-07T23:45:39.101 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 1 monclient: found mon.noname-b 2023-12-07T23:45:39.101 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:39.102 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:45:39.102 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.094+0000 7f608da57640 1 -- 172.21.15.129:0/2737131272 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60880aa9a0 con 0x7f60880a8930 2023-12-07T23:45:39.102 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.096+0000 7f608da57640 1 --2- 172.21.15.129:0/2737131272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f60880a8930 0x7f60880a8d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6084009870 tx=0x7f608402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecaf1ea907531cc4 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:39.104 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 1 -- 172.21.15.129:0/2737131272 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084031640 con 0x7f60880a8930 2023-12-07T23:45:39.104 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:39.104 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:45:39.104 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: dump: 2023-12-07T23:45:39.104 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: _finish_auth 0 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: _check_auth_tickets 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 1 -- 172.21.15.129:0/2737131272 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60840317a0 con 0x7f60880a8930 2023-12-07T23:45:39.105 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 1 -- 172.21.15.129:0/2737131272 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084031b30 con 0x7f60880a8930 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608ca55640 10 monclient: dump: 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:39.106 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608fcbb640 5 monclient: authenticate success, global_id 4172 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608fcbb640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.099+0000 7f608fcbb640 4 set_mon_vals no callback set 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.100+0000 7f608fcbb640 10 monclient: shutdown 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.100+0000 7f608fcbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.100+0000 7f608fcbb640 1 -- 172.21.15.129:0/2737131272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f60880a8930 msgr2=0x7f60880a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.100+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2737131272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f60880a8930 0x7f60880a8d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6084009870 tx=0x7f608402b8f0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.101+0000 7f608fcbb640 1 -- 172.21.15.129:0/2737131272 shutdown_connections 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.101+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2737131272 >> v2:172.21.15.184:3300/0 conn(0x7f608813f120 0x7f608813f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.101+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2737131272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f60880a8930 0x7f60880a8d10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.101+0000 7f608fcbb640 1 -- 172.21.15.129:0/2737131272 >> 172.21.15.129:0/2737131272 conn(0x7f60880b77a0 msgr2=0x7f60880b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.101+0000 7f608fcbb640 1 -- 172.21.15.129:0/2737131272 shutdown_connections 2023-12-07T23:45:39.107 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.101+0000 7f608fcbb640 1 -- 172.21.15.129:0/2737131272 wait complete. 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 10 monclient: build_initial_monmap 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 10 monclient: monmap: 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:last_changed 0.000000 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:created 0.000000 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:39.110 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 1 librados: starting msgr at 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 1 librados: starting objecter 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 1 Processor -- start 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 1 -- start start 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 1 librados: setting wanted keys 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 1 librados: calling monclient init 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.102+0000 7f608fcbb640 10 monclient: init 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60880a8930 0x7f6088139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): picked mon.noname-a con 0x7f60880a8930 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60880b21a0 0x7f608813ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): picked mon.noname-b con 0x7f60880b21a0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f608813f120 0x7f60881dab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): picked mon.noname-c con 0x7f608813f120 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:39.111 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60881da6f0 con 0x7f60880a8930 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60880abe50 con 0x7f60880b21a0 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6088137bf0 con 0x7f608813f120 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.103+0000 7f608fcbb640 10 monclient(hunting): authenticate will time out at 1339.291531s 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.104+0000 7f608d256640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60880b21a0 0x7f608813ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60880b21a0 0x7f608813ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:38584/0 (socket says 172.21.15.129:38584) 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 1 -- 172.21.15.129:0/2964437698 learned_addr learned my addr 172.21.15.129:0/2964437698 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 10 monclient(hunting): get_auth_request con 0x7f60880b21a0 auth_method 0 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:39.112 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608da57640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60880a8930 0x7f6088139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608da57640 10 monclient(hunting): get_auth_request con 0x7f60880a8930 auth_method 0 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608da57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608da57640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608da57640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.105+0000 7f608d256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608e258640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f608813f120 0x7f60881dab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608da57640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608da57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608da57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608e258640 10 monclient(hunting): get_auth_request con 0x7f608813f120 auth_method 0 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608e258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608e258640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.106+0000 7f608e258640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:39.113 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.107+0000 7f608e258640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:39.114 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.107+0000 7f608e258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:39.114 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.107+0000 7f608e258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 10 monclient(hunting): handle_auth_done global_id 4191 payload 824 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 1 -- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f608813f120 msgr2=0x7f60881dab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f608813f120 0x7f60881dab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 1 -- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60880b21a0 msgr2=0x7f608813ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60880b21a0 0x7f608813ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 10 monclient: _finish_hunting 0 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 1 monclient: found mon.noname-a 2023-12-07T23:45:39.115 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:39.116 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:45:39.116 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.109+0000 7f608da57640 1 -- 172.21.15.129:0/2964437698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840094d0 con 0x7f60880a8930 2023-12-07T23:45:39.116 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.112+0000 7f608da57640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60880a8930 0x7f6088139fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6084033000 tx=0x7f60840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:39.116 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.112+0000 7f607e7fc640 10 client.?.objecter ms_handle_connect 0x7f60880a8930 2023-12-07T23:45:39.116 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.112+0000 7f607e7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f607e7fc640 1 -- 172.21.15.129:0/2964437698 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608403ac60 con 0x7f60880a8930 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f607e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f607e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f607e7fc640 10 monclient: dump: 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:39.120 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f607e7fc640 10 monclient: _finish_auth 0 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f607e7fc640 10 monclient: _check_auth_tickets 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f607e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 5 monclient: authenticate success, global_id 4191 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 10 monclient: _renew_subs 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608813b280 con 0x7f60880a8930 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 10 monclient: _renew_subs 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60881db230 con 0x7f60880a8930 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.115+0000 7f608fcbb640 1 librados: init done 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.116+0000 7f608fcbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.116+0000 7f608fcbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:39.121 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.116+0000 7f608fcbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.116+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6088067b90 con 0x7f60880a8930 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.116+0000 7f607e7fc640 1 -- 172.21.15.129:0/2964437698 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6084031e10 con 0x7f60880a8930 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.116+0000 7f607e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.116+0000 7f607ffff640 4 set_mon_vals no callback set 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.117+0000 7f607e7fc640 1 -- 172.21.15.129:0/2964437698 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608404d450 con 0x7f60880a8930 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.117+0000 7f607e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.117+0000 7f607e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.117+0000 7f607e7fc640 10 monclient: dump: 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:39.122 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:39.123 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:39.123 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:39.123 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:39.123 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:39.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.182+0000 7f607e7fc640 1 -- 172.21.15.129:0/2964437698 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f608402f050 con 0x7f60880a8930 2023-12-07T23:45:39.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.182+0000 7f607e7fc640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f606c06c790 0x7f606c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:39.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.182+0000 7f608d256640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f606c06c790 0x7f606c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:39.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.182+0000 7f608d256640 10 monclient: get_auth_request con 0x7f606c06c790 auth_method 0 2023-12-07T23:45:39.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.183+0000 7f607e7fc640 10 client.4191.objecter ms_handle_connect 0x7f606c06c790 2023-12-07T23:45:39.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.183+0000 7f608d256640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f606c06c790 0x7f606c06ec50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f60881d8080 tx=0x7f6078009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 1 -- 172.21.15.129:0/2964437698 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f608407a720 con 0x7f60880a8930 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 10 client.4191.objecter ms_dispatch 0x7f6088000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 20 client.4191.objecter dump_active .. 0 homeless 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 1 -- 172.21.15.129:0/2964437698 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6084076a60 con 0x7f60880a8930 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:39.423 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.419+0000 7f607e7fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:39.576 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:17.541 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:39.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.619+0000 7f608fcbb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:39.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.619+0000 7f608fcbb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:39.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.619+0000 7f608fcbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:45:39.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.619+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f608813b5d0 con 0x7f60880a8930 2023-12-07T23:45:39.632 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:17.597 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:39.667 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.664+0000 7f607e7fc640 1 -- 172.21.15.129:0/2964437698 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6084035020 con 0x7f60880a8930 2023-12-07T23:45:39.667 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.664+0000 7f607e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:39.667 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.664+0000 7f607e7fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:39.668 INFO:teuthology.misc.health.smithi129.stdout: 2023-12-07T23:45:39.668 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:45:21.546459+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 10 librados: watch_flush enter 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 10 librados: watch_flush exit 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 20 client.4191.objecter shutdown clearing up homeless session... 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 10 client.4191.objecter successfully canceled tick 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f606c06c790 msgr2=0x7f606c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f606c06c790 0x7f606c06ec50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f60881d8080 tx=0x7f6078009040 comp rx=0 tx=0).stop 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 10 monclient: shutdown 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60880a8930 msgr2=0x7f6088139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60880a8930 0x7f6088139fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6084033000 tx=0x7f60840046c0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.670 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 shutdown_connections 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f606c06c790 0x7f606c06ec50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f608813f120 0x7f60881dab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60880b21a0 0x7f608813ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 --2- 172.21.15.129:0/2964437698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60880a8930 0x7f6088139fa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.666+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 >> 172.21.15.129:0/2964437698 conn(0x7f60880b77a0 msgr2=0x7f608813c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.667+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 shutdown_connections 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.667+0000 7f608fcbb640 1 -- 172.21.15.129:0/2964437698 wait complete. 2023-12-07T23:45:39.671 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:39.667+0000 7f608fcbb640 1 librados: shutdown 2023-12-07T23:45:39.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T23:45:40.026 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:17.995 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:40.081 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:18.040 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:40.112 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:18.069 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:40.119 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:18.093 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:40.219 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:18.190 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:40.525 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:18.492 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:44.849 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:22.815 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:44.853 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:22.819 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.220 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:23.191 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.224 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:23.195 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.339 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:23.304 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.343 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:23.308 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.408 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:23.377 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.414 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:23.383 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.827 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:23.785 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:45.832 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:23.789 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:46.077 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:24.052 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:46.083 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:24.058 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:46.154 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:24.114 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:46.158 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:24.118 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:46.518 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:24.485 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:46.522 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:24.489 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:46.545 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.642+0000 7f690c14e640 10 monclient: get_monmap_and_config 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.642+0000 7f690c14e640 10 monclient: build_initial_monmap 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.643+0000 7f690c14e640 10 monclient: monmap: 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:45:46.644131+0000 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:45:46.644131+0000 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:46.646 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:46.647 INFO:teuthology.misc.health.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:46.647 INFO:teuthology.misc.health.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:46.647 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:46.647 INFO:teuthology.misc.health.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:46.647 INFO:teuthology.misc.health.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:46.647 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.643+0000 7f690c14e640 1 Processor -- start 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.644+0000 7f690c14e640 1 -- start start 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.644+0000 7f690c14e640 10 monclient: init 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f69040a8990 0x7f69041459b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient(hunting): picked mon.noname-c con 0x7f69040a8990 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69040a8d70 addr v1:172.21.15.129:6789/0 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f690413f790 0x7f690414c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:46.649 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient(hunting): picked mon.noname-a con 0x7f690413f790 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:46.650 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6904145520 con 0x7f69040a8d70 2023-12-07T23:45:46.650 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:46.650 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69040aabe0 con 0x7f690413f790 2023-12-07T23:45:46.650 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:46.650 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69040aaec0 con 0x7f69040a8990 2023-12-07T23:45:46.650 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:46.650 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.645+0000 7f690c14e640 10 monclient(hunting): authenticate will time out at 1346.833131s 2023-12-07T23:45:46.678 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.675+0000 7f690a6eb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f690413f790 0x7f690414c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:46.678 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.675+0000 7f69096e9640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f69040a8d70 0x7f690413f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36102/0 (socket says 172.21.15.129:36102) 2023-12-07T23:45:46.678 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.675+0000 7f69096e9640 1 -- 172.21.15.129:0/3087904453 learned_addr learned my addr 172.21.15.129:0/3087904453 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:45:46.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.676+0000 7f690a6eb640 10 monclient(hunting): get_auth_request con 0x7f690413f790 auth_method 0 2023-12-07T23:45:46.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.676+0000 7f690a6eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:46.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.676+0000 7f690a6eb640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:46.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.676+0000 7f690a6eb640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:46.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.676+0000 7f690a6eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:46.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.676+0000 7f690a6eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:46.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.676+0000 7f690a6eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:46.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2023-12-07T23:45:46.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 1 -- 172.21.15.129:0/3087904453 >> v2:172.21.15.184:3300/0 conn(0x7f69040a8990 msgr2=0x7f69041459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:46.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 1 --2- 172.21.15.129:0/3087904453 >> v2:172.21.15.184:3300/0 conn(0x7f69040a8990 0x7f69041459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:46.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 1 -- 172.21.15.129:0/3087904453 >> v1:172.21.15.129:6789/0 conn(0x7f69040a8d70 legacy=0x7f690413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:46.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 10 monclient: _finish_hunting 0 2023-12-07T23:45:46.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 1 monclient: found mon.noname-a 2023-12-07T23:45:46.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:46.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:45:46.683 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.679+0000 7f690a6eb640 1 -- 172.21.15.129:0/3087904453 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040a97c0 con 0x7f690413f790 2023-12-07T23:45:46.690 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.687+0000 7f690a6eb640 1 --2- 172.21.15.129:0/3087904453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f690413f790 0x7f690414c6f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69040b5420 tx=0x7f68fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f7278590ae1bb8 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 1 -- 172.21.15.129:0/3087904453 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc02cc80 con 0x7f690413f790 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: dump: 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:46.695 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: _finish_auth 0 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: _check_auth_tickets 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 1 -- 172.21.15.129:0/3087904453 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc02cde0 con 0x7f690413f790 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 1 -- 172.21.15.129:0/3087904453 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc034660 con 0x7f690413f790 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f6908ee8640 10 monclient: dump: 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:46.696 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 5 monclient: authenticate success, global_id 4194 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:46.697 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 4 set_mon_vals no callback set 2023-12-07T23:45:46.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 10 monclient: shutdown 2023-12-07T23:45:46.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:46.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 1 -- 172.21.15.129:0/3087904453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f690413f790 msgr2=0x7f690414c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:46.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 1 --2- 172.21.15.129:0/3087904453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f690413f790 0x7f690414c6f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69040b5420 tx=0x7f68fc02bed0 comp rx=0 tx=0).stop 2023-12-07T23:45:46.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 1 -- 172.21.15.129:0/3087904453 shutdown_connections 2023-12-07T23:45:46.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 1 --2- 172.21.15.129:0/3087904453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f690413f790 0x7f690414c6f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 1 --2- 172.21.15.129:0/3087904453 >> v2:172.21.15.184:3300/0 conn(0x7f69040a8990 0x7f69041459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.692+0000 7f690c14e640 1 -- 172.21.15.129:0/3087904453 >> 172.21.15.129:0/3087904453 conn(0x7f6904066dd0 msgr2=0x7f69040b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.693+0000 7f690c14e640 1 -- 172.21.15.129:0/3087904453 shutdown_connections 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.693+0000 7f690c14e640 1 -- 172.21.15.129:0/3087904453 wait complete. 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 10 monclient: build_initial_monmap 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 10 monclient: monmap: 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:46.702 INFO:teuthology.misc.health.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:last_changed 0.000000 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:created 0.000000 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 1 librados: starting msgr at 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 1 librados: starting objecter 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 1 Processor -- start 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 1 -- start start 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 1 librados: setting wanted keys 2023-12-07T23:45:46.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 1 librados: calling monclient init 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.694+0000 7f690c14e640 10 monclient: init 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69040a8990 0x7f6904140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): picked mon.noname-c con 0x7f69040a8990 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69040a8d70 0x7f6904141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): picked mon.noname-a con 0x7f69040a8d70 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f690413f790 0x7f6904135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): picked mon.noname-b con 0x7f690413f790 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904141740 con 0x7f69040a8d70 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:46.704 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904141e80 con 0x7f690413f790 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6904141960 con 0x7f69040a8990 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f690c14e640 10 monclient(hunting): authenticate will time out at 1346.883436s 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f6909eea640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69040a8990 0x7f6904140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.695+0000 7f69096e9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69040a8d70 0x7f6904141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69040a8990 0x7f6904140a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55326/0 (socket says 172.21.15.129:55326) 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 1 -- 172.21.15.129:0/1155492932 learned_addr learned my addr 172.21.15.129:0/1155492932 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 10 monclient(hunting): get_auth_request con 0x7f69040a8990 auth_method 0 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f69096e9640 10 monclient(hunting): get_auth_request con 0x7f69040a8d70 auth_method 0 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f69096e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:46.705 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f69096e9640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:46.706 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f69096e9640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:46.706 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:46.706 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:46.706 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f6909eea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:46.706 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f69096e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:46.706 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f69096e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:46.706 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.696+0000 7f69096e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:46.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2023-12-07T23:45:46.707 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 1 -- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f690413f790 msgr2=0x7f6904135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f690413f790 0x7f6904135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 1 -- 172.21.15.129:0/1155492932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69040a8d70 msgr2=0x7f6904141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69040a8d70 0x7f6904141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 10 monclient: _finish_hunting 0 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 1 monclient: found mon.noname-c 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.699+0000 7f6909eea640 1 -- 172.21.15.129:0/1155492932 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc009a70 con 0x7f69040a8990 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.701+0000 7f6909eea640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69040a8990 0x7f6904140a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f690000b4d0 tx=0x7f690000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.701+0000 7f68fa7fc640 10 client.?.objecter ms_handle_connect 0x7f69040a8990 2023-12-07T23:45:46.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.701+0000 7f68fa7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:46.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 1 -- 172.21.15.129:0/1155492932 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900011830 con 0x7f69040a8990 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: dump: 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: _finish_auth 0 2023-12-07T23:45:46.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: _check_auth_tickets 2023-12-07T23:45:46.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:46.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 1 -- 172.21.15.129:0/1155492932 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6900011990 con 0x7f69040a8990 2023-12-07T23:45:46.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:46.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 1 -- 172.21.15.129:0/1155492932 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900011d40 con 0x7f69040a8990 2023-12-07T23:45:46.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:46.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:46.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fa7fc640 10 monclient: dump: 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.705+0000 7f68fbfff640 4 set_mon_vals no callback set 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 5 monclient: authenticate success, global_id 4178 2023-12-07T23:45:46.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 10 monclient: _renew_subs 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690413fc60 con 0x7f69040a8990 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 10 monclient: _renew_subs 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69041362d0 con 0x7f69040a8990 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.706+0000 7f690c14e640 1 librados: init done 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.707+0000 7f690c14e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.707+0000 7f690c14e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.707+0000 7f690c14e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:46.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.707+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690413fe50 con 0x7f69040a8990 2023-12-07T23:45:46.786 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.783+0000 7f68fa7fc640 1 -- 172.21.15.129:0/1155492932 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f690001a400 con 0x7f69040a8990 2023-12-07T23:45:46.787 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.783+0000 7f68fa7fc640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f68e806c7e0 0x7f68e806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:46.787 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.784+0000 7f69096e9640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f68e806c7e0 0x7f68e806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:46.787 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.784+0000 7f69096e9640 10 monclient: get_auth_request con 0x7f68e806c7e0 auth_method 0 2023-12-07T23:45:46.787 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.784+0000 7f69096e9640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f68e806c7e0 0x7f68e806eca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f69040b0b10 tx=0x7f68f4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:46.787 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.784+0000 7f68fa7fc640 10 client.4178.objecter ms_handle_connect 0x7f68e806c7e0 2023-12-07T23:45:46.974 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.971+0000 7f68fa7fc640 1 -- 172.21.15.129:0/1155492932 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f69000590f0 con 0x7f69040a8990 2023-12-07T23:45:46.974 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.971+0000 7f68fa7fc640 10 client.4178.objecter ms_dispatch 0x7f6904000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T23:45:46.974 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.971+0000 7f68fa7fc640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T23:45:46.975 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.971+0000 7f68fa7fc640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2023-12-07T23:45:46.975 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:46.971+0000 7f68fa7fc640 20 client.4178.objecter dump_active .. 0 homeless 2023-12-07T23:45:47.013 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.010+0000 7f68fa7fc640 1 -- 172.21.15.129:0/1155492932 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6900054d30 con 0x7f69040a8990 2023-12-07T23:45:47.014 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.011+0000 7f68fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:47.014 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.011+0000 7f68fa7fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:47.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.193+0000 7f690c14e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:47.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.193+0000 7f690c14e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:47.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.193+0000 7f690c14e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:47.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.193+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f690413a510 con 0x7f69040a8990 2023-12-07T23:45:47.263 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.260+0000 7f68fa7fc640 1 -- 172.21.15.129:0/1155492932 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6900015020 con 0x7f69040a8990 2023-12-07T23:45:47.263 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.260+0000 7f68fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:47.263 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.260+0000 7f68fa7fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:47.264 INFO:teuthology.misc.health.smithi129.stdout: 2023-12-07T23:45:47.264 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:45:21.546459+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.262+0000 7f690c14e640 10 librados: watch_flush enter 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.262+0000 7f690c14e640 10 librados: watch_flush exit 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.262+0000 7f690c14e640 20 client.4178.objecter shutdown clearing up homeless session... 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.262+0000 7f690c14e640 10 client.4178.objecter successfully canceled tick 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f68e806c7e0 msgr2=0x7f68e806eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f68e806c7e0 0x7f68e806eca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f69040b0b10 tx=0x7f68f4009040 comp rx=0 tx=0).stop 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 10 monclient: shutdown 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69040a8990 msgr2=0x7f6904140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69040a8990 0x7f6904140a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f690000b4d0 tx=0x7f690000b990 comp rx=0 tx=0).stop 2023-12-07T23:45:47.266 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 shutdown_connections 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f68e806c7e0 0x7f68e806eca0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f690413f790 0x7f6904135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69040a8d70 0x7f6904141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 --2- 172.21.15.129:0/1155492932 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69040a8990 0x7f6904140a20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 >> 172.21.15.129:0/1155492932 conn(0x7f6904066dd0 msgr2=0x7f690413c150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 shutdown_connections 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 -- 172.21.15.129:0/1155492932 wait complete. 2023-12-07T23:45:47.267 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:47.263+0000 7f690c14e640 1 librados: shutdown 2023-12-07T23:45:47.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T23:45:49.116 INFO:tasks.ceph.osd.6.smithi184.stderr:==00:00:00:27.081 87203== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:49.245 INFO:tasks.ceph.osd.4.smithi184.stderr:==00:00:00:27.216 87201== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:49.453 INFO:tasks.ceph.osd.7.smithi184.stderr:==00:00:00:27.418 87204== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:49.705 INFO:tasks.ceph.osd.5.smithi184.stderr:==00:00:00:27.674 87202== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:50.218 INFO:tasks.ceph.osd.1.smithi129.stderr:==00:00:00:28.175 87056== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:50.671 INFO:tasks.ceph.osd.0.smithi129.stderr:==00:00:00:28.646 87053== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:50.722 INFO:tasks.ceph.osd.2.smithi129.stderr:==00:00:00:28.681 87055== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:50.922 INFO:tasks.ceph.osd.3.smithi129.stderr:==00:00:00:28.889 87054== Warning: unimplemented fcntl command: 1038 2023-12-07T23:45:54.088 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.175+0000 7f6448b55640 10 monclient: get_monmap_and_config 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.175+0000 7f6448b55640 10 monclient: build_initial_monmap 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.175+0000 7f6448b55640 10 monclient: monmap: 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:45:54.176969+0000 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:45:54.176969+0000 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:45:54.179 INFO:teuthology.misc.health.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:45:54.180 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:45:54.180 INFO:teuthology.misc.health.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:45:54.180 INFO:teuthology.misc.health.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:45:54.180 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:54.180 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.176+0000 7f6448b55640 1 Processor -- start 2023-12-07T23:45:54.180 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.176+0000 7f6448b55640 1 -- start start 2023-12-07T23:45:54.180 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.176+0000 7f6448b55640 10 monclient: init 2023-12-07T23:45:54.182 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f64440a8930 0x7f64440a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient(hunting): picked mon.noname-c con 0x7f64440a8930 addr v2:172.21.15.184:3300/0 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64440b21a0 addr v1:172.21.15.184:6789/0 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f644413f120 0x7f644413f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient(hunting): picked mon.noname-a con 0x7f644413f120 addr v2:172.21.15.129:3300/0 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64440b1c20 con 0x7f64440b21a0 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64440b25e0 con 0x7f644413f120 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64440b1e70 con 0x7f64440a8930 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.177+0000 7f6448b55640 10 monclient(hunting): authenticate will time out at 1354.365690s 2023-12-07T23:45:54.183 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f6442d9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f644413f120 0x7f644413f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f644259c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f64440a8930 0x7f64440a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f6442d9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f644413f120 0x7f644413f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42286/0 (socket says 172.21.15.129:42286) 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f6442d9d640 1 -- 172.21.15.129:0/1191024368 learned_addr learned my addr 172.21.15.129:0/1191024368 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f6442d9d640 10 monclient(hunting): get_auth_request con 0x7f644413f120 auth_method 0 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f6442d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f6442d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.178+0000 7f6442d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.179+0000 7f644259c640 10 monclient(hunting): get_auth_request con 0x7f64440a8930 auth_method 0 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.179+0000 7f644259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.179+0000 7f644259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.179+0000 7f644259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.179+0000 7f6442d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.179+0000 7f6442d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.179+0000 7f6442d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.180+0000 7f644259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:54.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.180+0000 7f644259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:54.185 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.180+0000 7f644259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 1 -- 172.21.15.129:0/1191024368 >> v2:172.21.15.184:3300/0 conn(0x7f64440a8930 msgr2=0x7f64440a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 1 --2- 172.21.15.129:0/1191024368 >> v2:172.21.15.184:3300/0 conn(0x7f64440a8930 0x7f64440a8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 1 -- 172.21.15.129:0/1191024368 >> v1:172.21.15.184:6789/0 conn(0x7f64440b21a0 legacy=0x7f6444142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 10 monclient: _finish_hunting 0 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 1 monclient: found mon.noname-a 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.181+0000 7f6442d9d640 1 -- 172.21.15.129:0/1191024368 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440aa9a0 con 0x7f644413f120 2023-12-07T23:45:54.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.183+0000 7f6442d9d640 1 --2- 172.21.15.129:0/1191024368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 0x7f644413f550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f64440a8650 tx=0x7f643c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd7fb80d22276f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 1 -- 172.21.15.129:0/1191024368 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c02ca80 con 0x7f644413f120 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: dump: 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.190 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: _finish_auth 0 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: _check_auth_tickets 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 1 -- 172.21.15.129:0/1191024368 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c02cbe0 con 0x7f644413f120 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 1 -- 172.21.15.129:0/1191024368 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0343f0 con 0x7f644413f120 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f644159a640 10 monclient: dump: 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:54.191 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f6448b55640 5 monclient: authenticate success, global_id 4203 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f6448b55640 10 monclient: get_monmap_and_config success 2023-12-07T23:45:54.192 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.187+0000 7f6448b55640 4 set_mon_vals no callback set 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 10 monclient: shutdown 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 -- 172.21.15.129:0/1191024368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 msgr2=0x7f644413f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 --2- 172.21.15.129:0/1191024368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 0x7f644413f550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f64440a8650 tx=0x7f643c02bee0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 -- 172.21.15.129:0/1191024368 shutdown_connections 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 --2- 172.21.15.129:0/1191024368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 0x7f644413f550 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 --2- 172.21.15.129:0/1191024368 >> v2:172.21.15.184:3300/0 conn(0x7f64440a8930 0x7f64440a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 -- 172.21.15.129:0/1191024368 >> 172.21.15.129:0/1191024368 conn(0x7f64440b77a0 msgr2=0x7f64440b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 -- 172.21.15.129:0/1191024368 shutdown_connections 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.188+0000 7f6448b55640 1 -- 172.21.15.129:0/1191024368 wait complete. 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.189+0000 7f6448b55640 10 monclient: build_initial_monmap 2023-12-07T23:45:54.197 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.189+0000 7f6448b55640 10 monclient: monmap: 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:last_changed 0.000000 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:created 0.000000 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.189+0000 7f6448b55640 1 librados: starting msgr at 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.189+0000 7f6448b55640 1 librados: starting objecter 2023-12-07T23:45:54.198 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.190+0000 7f6448b55640 1 Processor -- start 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.190+0000 7f6448b55640 1 -- start start 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.190+0000 7f6448b55640 1 librados: setting wanted keys 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.190+0000 7f6448b55640 1 librados: calling monclient init 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.190+0000 7f6448b55640 10 monclient: init 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.190+0000 7f6448b55640 10 monclient: _reopen_session rank -1 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64440a8930 0x7f6444137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): picked mon.noname-b con 0x7f64440a8930 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64440b21a0 0x7f6444145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): picked mon.noname-c con 0x7f64440b21a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 0x7f6444133880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): picked mon.noname-a con 0x7f644413f120 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:54.199 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6444138fd0 con 0x7f644413f120 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6444146000 con 0x7f64440a8930 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): start opening mon connection 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6444145b50 con 0x7f64440b21a0 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): _renew_subs 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6448b55640 10 monclient(hunting): authenticate will time out at 1354.379191s 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f6441d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64440b21a0 0x7f6444145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.191+0000 7f644259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64440a8930 0x7f6444137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64440b21a0 0x7f6444145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:41054/0 (socket says 172.21.15.129:41054) 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 1 -- 172.21.15.129:0/4274987095 learned_addr learned my addr 172.21.15.129:0/4274987095 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 10 monclient(hunting): get_auth_request con 0x7f64440b21a0 auth_method 0 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:54.200 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f644259c640 10 monclient(hunting): get_auth_request con 0x7f64440a8930 auth_method 0 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f644259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f644259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f644259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6442d9d640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 0x7f6444133880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6442d9d640 10 monclient(hunting): get_auth_request con 0x7f644413f120 auth_method 0 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6442d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6442d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6442d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:54.201 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f6441d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:54.202 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f644259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:54.202 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f644259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:54.202 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.192+0000 7f644259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:54.202 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.193+0000 7f6442d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:45:54.202 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.193+0000 7f6442d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:45:54.202 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.193+0000 7f6442d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:45:54.203 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 10 monclient(hunting): handle_auth_done global_id 4181 payload 824 2023-12-07T23:45:54.203 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 1 -- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64440a8930 msgr2=0x7f6444137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:54.203 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64440a8930 0x7f6444137d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.203 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 1 -- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 msgr2=0x7f6444133880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 0x7f6444133880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 10 monclient: _finish_hunting 0 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 1 monclient: found mon.noname-c 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.196+0000 7f6441d9b640 1 -- 172.21.15.129:0/4274987095 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c009b30 con 0x7f64440b21a0 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.199+0000 7f6441d9b640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64440b21a0 0x7f6444145610 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f643800c400 tx=0x7f643800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.199+0000 7f6432ffd640 10 client.?.objecter ms_handle_connect 0x7f64440b21a0 2023-12-07T23:45:54.204 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.199+0000 7f6432ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:45:54.209 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.204+0000 7f6432ffd640 1 -- 172.21.15.129:0/4274987095 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64380052d0 con 0x7f64440b21a0 2023-12-07T23:45:54.209 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.204+0000 7f6432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:54.209 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.204+0000 7f6432ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:45:54.209 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.204+0000 7f6432ffd640 10 monclient: dump: 2023-12-07T23:45:54.209 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:54.209 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:54.209 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.204+0000 7f6432ffd640 10 monclient: _finish_auth 0 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.204+0000 7f6432ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.204+0000 7f6432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 5 monclient: authenticate success, global_id 4181 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 10 monclient: _renew_subs 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6444145dd0 con 0x7f64440b21a0 2023-12-07T23:45:54.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 10 monclient: _renew_subs 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6444133fa0 con 0x7f64440b21a0 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.205+0000 7f6448b55640 1 librados: init done 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6432ffd640 1 -- 172.21.15.129:0/4274987095 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6438005430 con 0x7f64440b21a0 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6432ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6432ffd640 1 -- 172.21.15.129:0/4274987095 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438007220 con 0x7f64440b21a0 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6432ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6432ffd640 10 monclient: dump: 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:45:54.211 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6440d99640 4 set_mon_vals no callback set 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.206+0000 7f6448b55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.207+0000 7f6448b55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:54.212 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.207+0000 7f6448b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:54.213 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.207+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6428005080 con 0x7f64440b21a0 2023-12-07T23:45:54.230 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.226+0000 7f6432ffd640 1 -- 172.21.15.129:0/4274987095 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f6438007380 con 0x7f64440b21a0 2023-12-07T23:45:54.230 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.226+0000 7f6432ffd640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f642406c790 0x7f642406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:45:54.230 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.227+0000 7f644259c640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f642406c790 0x7f642406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:45:54.230 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.227+0000 7f644259c640 10 monclient: get_auth_request con 0x7f642406c790 auth_method 0 2023-12-07T23:45:54.230 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.227+0000 7f6432ffd640 10 client.4181.objecter ms_handle_connect 0x7f642406c790 2023-12-07T23:45:54.231 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.227+0000 7f644259c640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f642406c790 0x7f642406ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6434009aa0 tx=0x7f6434009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:45:54.479 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 1 -- 172.21.15.129:0/4274987095 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6438058040 con 0x7f64440b21a0 2023-12-07T23:45:54.479 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 10 client.4181.objecter ms_dispatch 0x7f6444000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T23:45:54.479 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T23:45:54.479 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2023-12-07T23:45:54.479 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 20 client.4181.objecter dump_active .. 0 homeless 2023-12-07T23:45:54.479 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 1 -- 172.21.15.129:0/4274987095 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6438054380 con 0x7f64440b21a0 2023-12-07T23:45:54.479 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:45:54.480 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.476+0000 7f6432ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:45:54.746 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.743+0000 7f6448b55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:54.746 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.743+0000 7f6448b55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:54.746 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.743+0000 7f6448b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:45:54.746 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.743+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6428005600 con 0x7f64440b21a0 2023-12-07T23:45:54.760 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.757+0000 7f6432ffd640 1 -- 172.21.15.129:0/4274987095 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6438020090 con 0x7f64440b21a0 2023-12-07T23:45:54.761 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.757+0000 7f6432ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:45:54.761 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.757+0000 7f6432ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:45:54.761 INFO:teuthology.misc.health.smithi129.stdout: 2023-12-07T23:45:54.761 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:45:21.546459+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 10 librados: watch_flush enter 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 10 librados: watch_flush exit 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 20 client.4181.objecter shutdown clearing up homeless session... 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 10 client.4181.objecter successfully canceled tick 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f642406c790 msgr2=0x7f642406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f642406c790 0x7f642406ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6434009aa0 tx=0x7f6434009190 comp rx=0 tx=0).stop 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 10 monclient: shutdown 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:45:54.764 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64440b21a0 msgr2=0x7f6444145610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64440b21a0 0x7f6444145610 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f643800c400 tx=0x7f643800c8c0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 shutdown_connections 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f642406c790 0x7f642406ec50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644413f120 0x7f6444133880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64440b21a0 0x7f6444145610 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 --2- 172.21.15.129:0/4274987095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64440a8930 0x7f6444137d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 >> 172.21.15.129:0/4274987095 conn(0x7f64440b77a0 msgr2=0x7f64440b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 shutdown_connections 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 -- 172.21.15.129:0/4274987095 wait complete. 2023-12-07T23:45:54.765 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:45:54.760+0000 7f6448b55640 1 librados: shutdown 2023-12-07T23:45:54.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T23:45:57.228 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:45:57.222+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-12-07T23:45:57.340 INFO:tasks.ceph.osd.7.smithi184.stderr:2023-12-07T23:45:57.334+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-12-07T23:45:57.452 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:45:57.446+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-12-07T23:45:58.047 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:45:58.042+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-12-07T23:45:58.544 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:45:58.538+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-12-07T23:45:59.253 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:45:59.246+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-12-07T23:45:59.316 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:45:59.310+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-12-07T23:45:59.439 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:45:59.432+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-12-07T23:46:00.680 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:46:00.670+0000 ac7d640 -1 osd.5 0 waiting for initial osdmap 2023-12-07T23:46:00.725 INFO:tasks.ceph.osd.7.smithi184.stderr:2023-12-07T23:46:00.717+0000 c493640 -1 osd.7 0 waiting for initial osdmap 2023-12-07T23:46:00.740 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:46:00.728+0000 c493640 -1 osd.6 0 waiting for initial osdmap 2023-12-07T23:46:00.764 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:46:00.755+0000 c493640 -1 osd.4 0 waiting for initial osdmap 2023-12-07T23:46:01.114 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:46:01.096+0000 10253640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:01.143 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:46:01.121+0000 10253640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:01.164 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:46:01.147+0000 10253640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:01.214 INFO:tasks.ceph.osd.7.smithi184.stderr:2023-12-07T23:46:01.197+0000 10253640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:01.585 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T23:46:01.679 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.676+0000 7f9507cec640 10 monclient: get_monmap_and_config 2023-12-07T23:46:01.680 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.676+0000 7f9507cec640 10 monclient: build_initial_monmap 2023-12-07T23:46:01.680 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.676+0000 7f9507cec640 10 monclient: monmap: 2023-12-07T23:46:01.680 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:46:01.680 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:01.680 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:46:01.677280+0000 2023-12-07T23:46:01.680 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:46:01.677280+0000 2023-12-07T23:46:01.681 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:01.681 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:01.681 INFO:teuthology.misc.health.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:01.681 INFO:teuthology.misc.health.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:01.681 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:01.681 INFO:teuthology.misc.health.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:01.681 INFO:teuthology.misc.health.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:01.682 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:01.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.676+0000 7f9507cec640 1 Processor -- start 2023-12-07T23:46:01.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.676+0000 7f9507cec640 1 -- start start 2023-12-07T23:46:01.682 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.676+0000 7f9507cec640 10 monclient: init 2023-12-07T23:46:01.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:01.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-12-07T23:46:01.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95000a8930 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:01.684 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95000a8d10 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:01.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f950013f830 0x7f95000b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient(hunting): picked mon.noname-b con 0x7f950013f830 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:01.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000b1c20 con 0x7f95000a8d10 2023-12-07T23:46:01.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000b1e70 con 0x7f95000a8930 2023-12-07T23:46:01.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:01.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95000aa9a0 con 0x7f950013f830 2023-12-07T23:46:01.685 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:01.686 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.677+0000 7f9507cec640 10 monclient(hunting): authenticate will time out at 1361.865556s 2023-12-07T23:46:01.699 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.696+0000 7f9505287640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f95000a8d10 0x7f950013f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58710/0 (socket says 172.21.15.129:58710) 2023-12-07T23:46:01.699 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.696+0000 7f9505287640 1 -- 172.21.15.129:0/4264812676 learned_addr learned my addr 172.21.15.129:0/4264812676 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:01.701 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.698+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145396230 0 0) 0x7f95000b1c20 con 0x7f95000a8d10 2023-12-07T23:46:01.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.698+0000 7f9504a86640 10 monclient(hunting): _negotiate 2023-12-07T23:46:01.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.698+0000 7f9504a86640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:01.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.698+0000 7f9504a86640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:01.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.698+0000 7f9504a86640 10 monclient(hunting): my global_id is 4221 2023-12-07T23:46:01.702 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.698+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec003520 con 0x7f95000a8d10 2023-12-07T23:46:01.703 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.700+0000 7f9506289640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f950013f830 0x7f95000b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:01.708 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3425623094 0 0) 0x7f94ec003520 con 0x7f95000a8d10 2023-12-07T23:46:01.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f950013f830 msgr2=0x7f95000b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:01.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f950013f830 0x7f95000b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:01.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 >> v1:172.21.15.184:6789/0 conn(0x7f95000a8930 legacy=0x7f9500142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:01.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: _finish_hunting 0 2023-12-07T23:46:01.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 monclient: found mon.noname-a-legacy 2023-12-07T23:46:01.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:01.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.129:6789/0 2023-12-07T23:46:01.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9500142e40 con 0x7f95000a8d10 2023-12-07T23:46:01.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: _finish_auth 0 2023-12-07T23:46:01.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: _check_auth_tickets 2023-12-07T23:46:01.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:01.710 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 314329121 0 0) 0x7f94f0001670 con 0x7f95000a8d10 2023-12-07T23:46:01.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9507cec640 5 monclient: authenticate success, global_id 4221 2023-12-07T23:46:01.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9507cec640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:01.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:01.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-07T23:46:01.711 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: dump: 2023-12-07T23:46:01.711 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:01.712 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:01.712 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.712 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.712 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:01.712 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:01.712 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:01.713 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:01.713 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:01.713 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:01.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] but i'm connected to v1:172.21.15.129:6789/0, reconnecting 2023-12-07T23:46:01.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:01.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 >> v1:172.21.15.129:6789/0 conn(0x7f95000a8d10 legacy=0x7f950013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:01.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:01.714 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94ec0039a0 0x7f94ec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.714 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): picked mon.b con 0x7f94ec0039a0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:01.714 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94ec0047c0 0x7f94ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.715 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): picked mon.a con 0x7f94ec0047c0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:01.715 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f950013f830 0x7f94ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.715 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): picked mon.c con 0x7f950013f830 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:01.715 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:01.715 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95000aa9a0 con 0x7f94ec0047c0 2023-12-07T23:46:01.715 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:01.716 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec0054e0 con 0x7f94ec0039a0 2023-12-07T23:46:01.716 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:01.716 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec00bec0 con 0x7f950013f830 2023-12-07T23:46:01.716 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:01.716 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9504a86640 10 monclient(hunting): _finish_auth 0 2023-12-07T23:46:01.716 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.702+0000 7f9507cec640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:01.717 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.703+0000 7f9505287640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94ec0047c0 0x7f94ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:01.717 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.704+0000 7f9505287640 10 monclient(hunting): get_auth_request con 0x7f94ec0047c0 auth_method 0 2023-12-07T23:46:01.717 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.704+0000 7f9505287640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:01.717 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.704+0000 7f9505287640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:01.717 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.704+0000 7f9505287640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T23:46:01.727 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.706+0000 7f9505287640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:01.728 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.706+0000 7f9505287640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:01.728 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.706+0000 7f9505287640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T23:46:01.728 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 10 monclient(hunting): handle_auth_done global_id 4221 payload 293 2023-12-07T23:46:01.728 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 1 -- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f950013f830 msgr2=0x7f94ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:01.728 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f950013f830 0x7f94ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:01.728 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 1 -- 172.21.15.129:0/4264812676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94ec0039a0 msgr2=0x7f94ec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:01.728 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94ec0039a0 0x7f94ec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 10 monclient: _finish_hunting 0 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 1 monclient: found mon.a 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 1 -- 172.21.15.129:0/4264812676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec003520 con 0x7f94ec0047c0 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 10 monclient: _finish_auth 0 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 10 monclient: _check_auth_tickets 2023-12-07T23:46:01.729 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.708+0000 7f9505287640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:01.730 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.710+0000 7f9505287640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94ec0047c0 0x7f94ec004c40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f94f000c0e0 tx=0x7f94ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fad9e8220666f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:01.730 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0005ad0 con 0x7f94ec0047c0 2023-12-07T23:46:01.730 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:01.730 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 10 monclient: dump: 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f0005c30 con 0x7f94ec0047c0 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 1 -- 172.21.15.129:0/4264812676 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f002ca50 con 0x7f94ec0047c0 2023-12-07T23:46:01.731 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9507cec640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9507cec640 4 set_mon_vals no callback set 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.714+0000 7f9504a86640 10 monclient: dump: 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:01.732 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 10 monclient: shutdown 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 1 -- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94ec0047c0 msgr2=0x7f94ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 1 --2- 172.21.15.129:0/4264812676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94ec0047c0 0x7f94ec004c40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f94f000c0e0 tx=0x7f94ec0057f0 comp rx=0 tx=0).stop 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9506289640 1 -- 172.21.15.129:0/4264812676 reap_dead start 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 1 -- 172.21.15.129:0/4264812676 shutdown_connections 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 1 -- 172.21.15.129:0/4264812676 >> 172.21.15.129:0/4264812676 conn(0x7f95000b77a0 msgr2=0x7f95000b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 1 -- 172.21.15.129:0/4264812676 shutdown_connections 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.715+0000 7f9507cec640 1 -- 172.21.15.129:0/4264812676 wait complete. 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.716+0000 7f9507cec640 10 monclient: build_initial_monmap 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.716+0000 7f9507cec640 10 monclient: monmap: 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:created 0.000000 2023-12-07T23:46:01.733 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.716+0000 7f9507cec640 1 librados: starting msgr at 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.716+0000 7f9507cec640 1 librados: starting objecter 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.716+0000 7f9507cec640 1 Processor -- start 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.716+0000 7f9507cec640 1 -- start start 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 1 librados: setting wanted keys 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 1 librados: calling monclient init 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 10 monclient: init 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:01.734 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94ec0047c0 0x7f95000ab580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 10 monclient(hunting): picked mon.noname-c con 0x7f94ec0047c0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9500143240 0x7f9500145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.717+0000 7f9507cec640 10 monclient(hunting): picked mon.noname-a con 0x7f9500143240 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95001d7330 0x7f95001d77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 10 monclient(hunting): picked mon.noname-b con 0x7f95001d7330 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95000aca30 con 0x7f9500143240 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9500145c70 con 0x7f95001d7330 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:01.735 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95000acc30 con 0x7f94ec0047c0 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9507cec640 10 monclient(hunting): authenticate will time out at 1361.906018s 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9505287640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9500143240 0x7f9500145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9505287640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9500143240 0x7f9500145730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42326/0 (socket says 172.21.15.129:42326) 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9505287640 1 -- 172.21.15.129:0/2422405728 learned_addr learned my addr 172.21.15.129:0/2422405728 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9505287640 10 monclient(hunting): get_auth_request con 0x7f9500143240 auth_method 0 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9505287640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9505287640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.718+0000 7f9505287640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.719+0000 7f9505287640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.719+0000 7f9505287640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.719+0000 7f9505287640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.720+0000 7f9506289640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95001d7330 0x7f95001d77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.720+0000 7f9506289640 10 monclient(hunting): get_auth_request con 0x7f95001d7330 auth_method 0 2023-12-07T23:46:01.736 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.720+0000 7f9506289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.720+0000 7f9506289640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.720+0000 7f9506289640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.721+0000 7f9506289640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.721+0000 7f9506289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.721+0000 7f9506289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 1 -- 172.21.15.129:0/2422405728 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94ec0047c0 msgr2=0x7f95000ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94ec0047c0 0x7f95000ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 1 -- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95001d7330 msgr2=0x7f95001d77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95001d7330 0x7f95001d77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 10 monclient: _finish_hunting 0 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 1 monclient: found mon.noname-a 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:01.737 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.722+0000 7f9505287640 1 -- 172.21.15.129:0/2422405728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95001d6e10 con 0x7f9500143240 2023-12-07T23:46:01.739 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.725+0000 7f9505287640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9500143240 0x7f9500145730 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f94f000c0e0 tx=0x7f95000b3d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:01.739 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.725+0000 7f94f67fc640 10 client.?.objecter ms_handle_connect 0x7f9500143240 2023-12-07T23:46:01.739 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.725+0000 7f94f67fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:01.739 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 1 -- 172.21.15.129:0/2422405728 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f002cae0 con 0x7f9500143240 2023-12-07T23:46:01.739 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:01.739 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:01.739 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: dump: 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:01.740 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 1 -- 172.21.15.129:0/2422405728 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f002cc40 con 0x7f9500143240 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:01.741 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 1 -- 172.21.15.129:0/2422405728 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0007070 con 0x7f9500143240 2023-12-07T23:46:01.742 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:01.742 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:01.742 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f67fc640 10 monclient: dump: 2023-12-07T23:46:01.742 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:01.742 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:01.742 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.742 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:01.743 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:01.743 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:01.743 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:01.743 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:01.743 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:01.743 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:01.743 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 5 monclient: authenticate success, global_id 4224 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 10 monclient: _renew_subs 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9500137b10 con 0x7f9500143240 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f94f7fff640 4 set_mon_vals no callback set 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 10 monclient: _renew_subs 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:01.744 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9500137f70 con 0x7f9500143240 2023-12-07T23:46:01.745 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.729+0000 7f9507cec640 1 librados: init done 2023-12-07T23:46:01.745 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.731+0000 7f9507cec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:01.745 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.731+0000 7f9507cec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:01.745 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.731+0000 7f9507cec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:01.745 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.731+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9500067b90 con 0x7f9500143240 2023-12-07T23:46:01.762 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.758+0000 7f94f67fc640 1 -- 172.21.15.129:0/2422405728 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f94f0032070 con 0x7f9500143240 2023-12-07T23:46:01.762 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.758+0000 7f94f67fc640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f94e406c7c0 0x7f94e406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:01.762 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.758+0000 7f9505a88640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f94e406c7c0 0x7f94e406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:01.762 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.758+0000 7f9505a88640 10 monclient: get_auth_request con 0x7f94e406c7c0 auth_method 0 2023-12-07T23:46:01.762 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.759+0000 7f94f67fc640 10 client.4224.objecter ms_handle_connect 0x7f94e406c7c0 2023-12-07T23:46:01.762 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:01.759+0000 7f9505a88640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f94e406c7c0 0x7f94e406ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f94fc001550 tx=0x7f94fc007990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:01.931 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:46:01.925+0000 c493640 -1 osd.1 0 waiting for initial osdmap 2023-12-07T23:46:02.004 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 1 -- 172.21.15.129:0/2422405728 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3303+0+0 (secure 0 0 0) 0x7f94f007a180 con 0x7f9500143240 2023-12-07T23:46:02.004 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 10 client.4224.objecter ms_dispatch 0x7f9500000b90 osd_map(13..13 src has 1..13) v4 2023-12-07T23:46:02.004 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 3 client.4224.objecter handle_osd_map got epochs [13,13] > 0 2023-12-07T23:46:02.004 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 3 client.4224.objecter handle_osd_map decoding full epoch 13 2023-12-07T23:46:02.004 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 20 client.4224.objecter dump_active .. 0 homeless 2023-12-07T23:46:02.005 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 1 -- 172.21.15.129:0/2422405728 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94f00764c0 con 0x7f9500143240 2023-12-07T23:46:02.005 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:02.005 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.001+0000 7f94f67fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:02.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.193+0000 7f9507cec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:02.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.193+0000 7f9507cec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:02.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.193+0000 7f9507cec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:02.196 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.193+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9500067b90 con 0x7f9500143240 2023-12-07T23:46:02.255 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.252+0000 7f94f67fc640 1 -- 172.21.15.129:0/2422405728 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+8976 (secure 0 0 0) 0x7f94f004f070 con 0x7f9500143240 2023-12-07T23:46:02.255 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.252+0000 7f94f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:02.255 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.252+0000 7f94f67fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:02.256 INFO:teuthology.misc.health.smithi129.stdout: 2023-12-07T23:46:02.256 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":13,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:46:01.539004+0000","last_up_change":"2023-12-07T23:46:01.539004+0000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6827","nonce":917696104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6829","nonce":917696104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6833","nonce":917696104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6831","nonce":917696104}]},"public_addr":"172.21.15.184:6827/917696104","cluster_addr":"172.21.15.184:6829/917696104","heartbeat_back_addr":"172.21.15.184:6833/917696104","heartbeat_front_addr":"172.21.15.184:6831/917696104","state":["exists","up"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6819","nonce":4213359653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6821","nonce":4213359653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6825","nonce":4213359653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6823","nonce":4213359653}]},"public_addr":"172.21.15.184:6819/4213359653","cluster_addr":"172.21.15.184:6821/4213359653","heartbeat_back_addr":"172.21.15.184:6825/4213359653","heartbeat_front_addr":"172.21.15.184:6823/4213359653","state":["exists","up"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6807","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6809","nonce":105483051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6813","nonce":105483051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6817","nonce":105483051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6815","nonce":105483051}]},"public_addr":"172.21.15.184:6809/105483051","cluster_addr":"172.21.15.184:6813/105483051","heartbeat_back_addr":"172.21.15.184:6817/105483051","heartbeat_front_addr":"172.21.15.184:6815/105483051","state":["exists","up"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6803","nonce":59760990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6805","nonce":59760990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6811","nonce":59760990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6808","nonce":59760990}]},"public_addr":"172.21.15.184:6803/59760990","cluster_addr":"172.21.15.184:6805/59760990","heartbeat_back_addr":"172.21.15.184:6811/59760990","heartbeat_front_addr":"172.21.15.184:6808/59760990","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 10 librados: watch_flush enter 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 10 librados: watch_flush exit 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 20 client.4224.objecter shutdown clearing up homeless session... 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 10 client.4224.objecter successfully canceled tick 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f94e406c7c0 msgr2=0x7f94e406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f94e406c7c0 0x7f94e406ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f94fc001550 tx=0x7f94fc007990 comp rx=0 tx=0).stop 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 10 monclient: shutdown 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9500143240 msgr2=0x7f9500145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:02.258 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9500143240 0x7f9500145730 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f94f000c0e0 tx=0x7f95000b3d00 comp rx=0 tx=0).stop 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 shutdown_connections 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f94e406c7c0 0x7f94e406ec80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95001d7330 0x7f95001d77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9500143240 0x7f9500145730 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 --2- 172.21.15.129:0/2422405728 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f94ec0047c0 0x7f95000ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 >> 172.21.15.129:0/2422405728 conn(0x7f95000b77a0 msgr2=0x7f9500139590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 shutdown_connections 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.254+0000 7f9507cec640 1 -- 172.21.15.129:0/2422405728 wait complete. 2023-12-07T23:46:02.259 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:02.255+0000 7f9507cec640 1 librados: shutdown 2023-12-07T23:46:02.270 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-12-07T23:46:02.355 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:46:02.339+0000 10253640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:03.192 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:46:03.185+0000 c493640 -1 osd.2 0 waiting for initial osdmap 2023-12-07T23:46:03.204 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:46:03.198+0000 c493640 -1 osd.3 0 waiting for initial osdmap 2023-12-07T23:46:03.220 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:46:03.216+0000 c493640 -1 osd.0 0 waiting for initial osdmap 2023-12-07T23:46:03.759 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:46:03.743+0000 10253640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:03.822 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:46:03.804+0000 10253640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:03.833 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:46:03.812+0000 10253640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T23:46:09.076 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T23:46:09.166 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.163+0000 7f169a4c9640 10 monclient: get_monmap_and_config 2023-12-07T23:46:09.166 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.163+0000 7f169a4c9640 10 monclient: build_initial_monmap 2023-12-07T23:46:09.166 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.163+0000 7f169a4c9640 10 monclient: monmap: 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:46:09.164771+0000 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:46:09.164771+0000 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:09.167 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:09.169 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 1 Processor -- start 2023-12-07T23:46:09.169 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 1 -- start start 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 10 monclient: init 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16940a8990 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f16940a8d70 0x7f169413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f16940a8d70 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f169413f5c0 0x7f169414c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.164+0000 7f169a4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f169413f5c0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f169a4c9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1694145520 con 0x7f16940a8990 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f169a4c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f169a4c9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169413fa30 con 0x7f169413f5c0 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f169a4c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.170 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f169a4c9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16940aabe0 con 0x7f16940a8d70 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f169a4c9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f169a4c9640 10 monclient(hunting): authenticate will time out at 1369.352996s 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f16937fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f16940a8d70 0x7f169413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f1698a66640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f169413f5c0 0x7f169414c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f1693fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f16940a8990 0x7f16941459b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:39124/0 (socket says 172.21.15.129:39124) 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.165+0000 7f1693fff640 1 -- 172.21.15.129:0/1528807755 learned_addr learned my addr 172.21.15.129:0/1528807755 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f1698a66640 10 monclient(hunting): get_auth_request con 0x7f169413f5c0 auth_method 0 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f1698a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f1698a66640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f1698a66640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f16937fe640 10 monclient(hunting): get_auth_request con 0x7f16940a8d70 auth_method 0 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f16937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f16937fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f16937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.171 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f16937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.172 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f16937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.172 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f16937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.172 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f1698a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.172 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f1698a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.172 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.166+0000 7f1698a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.168+0000 7f1692ffd640 1 -- 172.21.15.129:0/1528807755 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757935280 0 0) 0x7f1694145520 con 0x7f16940a8990 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.168+0000 7f1692ffd640 10 monclient(hunting): _negotiate 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.168+0000 7f1692ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.168+0000 7f1692ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.168+0000 7f1692ffd640 10 monclient(hunting): my global_id is 4261 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.168+0000 7f1692ffd640 1 -- 172.21.15.129:0/1528807755 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c003520 con 0x7f16940a8990 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 1 -- 172.21.15.129:0/1528807755 >> v2:172.21.15.129:3300/0 conn(0x7f169413f5c0 msgr2=0x7f169414c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 1 --2- 172.21.15.129:0/1528807755 >> v2:172.21.15.129:3300/0 conn(0x7f169413f5c0 0x7f169414c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 1 -- 172.21.15.129:0/1528807755 >> v1:172.21.15.184:6789/0 conn(0x7f16940a8990 legacy=0x7f16941459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.173 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:09.174 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 1 monclient: found mon.noname-b 2023-12-07T23:46:09.174 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:09.174 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:09.174 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.169+0000 7f16937fe640 1 -- 172.21.15.129:0/1528807755 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16940aaec0 con 0x7f16940a8d70 2023-12-07T23:46:09.174 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.171+0000 7f16937fe640 1 --2- 172.21.15.129:0/1528807755 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f16940a8d70 0x7f169413f080 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16880097a0 tx=0x7f168802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfa454251c92a72c server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:09.177 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.174+0000 7f1692ffd640 1 -- 172.21.15.129:0/1528807755 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688031830 con 0x7f16940a8d70 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.174+0000 7f1692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.174+0000 7f1692ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.174+0000 7f1692ffd640 10 monclient: dump: 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.178 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 1 -- 172.21.15.129:0/1528807755 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1688031990 con 0x7f16940a8d70 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 1 -- 172.21.15.129:0/1528807755 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688031d20 con 0x7f16940a8d70 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f169a4c9640 5 monclient: authenticate success, global_id 4211 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f169a4c9640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f169a4c9640 4 set_mon_vals no callback set 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f1692ffd640 10 monclient: dump: 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.179 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.180 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.180 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.180 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.180 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.180 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.180 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f169a4c9640 10 monclient: shutdown 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f169a4c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f169a4c9640 1 -- 172.21.15.129:0/1528807755 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f16940a8d70 msgr2=0x7f169413f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.175+0000 7f169a4c9640 1 --2- 172.21.15.129:0/1528807755 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f16940a8d70 0x7f169413f080 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16880097a0 tx=0x7f168802b8d0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.176+0000 7f169a4c9640 1 -- 172.21.15.129:0/1528807755 shutdown_connections 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.176+0000 7f169a4c9640 1 --2- 172.21.15.129:0/1528807755 >> v2:172.21.15.129:3300/0 conn(0x7f169413f5c0 0x7f169414c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.176+0000 7f169a4c9640 1 --2- 172.21.15.129:0/1528807755 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f16940a8d70 0x7f169413f080 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.176+0000 7f169a4c9640 1 -- 172.21.15.129:0/1528807755 >> 172.21.15.129:0/1528807755 conn(0x7f1694066dd0 msgr2=0x7f16940b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.176+0000 7f169a4c9640 1 -- 172.21.15.129:0/1528807755 shutdown_connections 2023-12-07T23:46:09.184 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.176+0000 7f169a4c9640 1 -- 172.21.15.129:0/1528807755 wait complete. 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 10 monclient: build_initial_monmap 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 10 monclient: monmap: 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:epoch 0 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:created 0.000000 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 1 librados: starting msgr at 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 1 librados: starting objecter 2023-12-07T23:46:09.185 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 1 Processor -- start 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 1 -- start start 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 1 librados: setting wanted keys 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 1 librados: calling monclient init 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.177+0000 7f169a4c9640 10 monclient: init 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16940a8990 0x7f1694143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f16940a8990 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16940a8d70 0x7f16941436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f16940a8d70 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f169413f5c0 0x7f1694135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f169413f5c0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.186 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1694144c10 con 0x7f16940a8d70 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169413fe20 con 0x7f16940a8990 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1694143dc0 con 0x7f169413f5c0 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.178+0000 7f169a4c9640 10 monclient(hunting): authenticate will time out at 1369.366866s 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1693fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16940a8990 0x7f1694143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1698a66640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f169413f5c0 0x7f1694135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f16937fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16940a8d70 0x7f16941436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1693fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16940a8990 0x7f1694143190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:51198/0 (socket says 172.21.15.129:51198) 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1693fff640 1 -- 172.21.15.129:0/3506589981 learned_addr learned my addr 172.21.15.129:0/3506589981 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1693fff640 10 monclient(hunting): get_auth_request con 0x7f16940a8990 auth_method 0 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1693fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.187 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1693fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1698a66640 10 monclient(hunting): get_auth_request con 0x7f169413f5c0 auth_method 0 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1698a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1698a66640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.179+0000 7f1698a66640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f16937fe640 10 monclient(hunting): get_auth_request con 0x7f16940a8d70 auth_method 0 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f16937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f16937fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f16937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f1693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f1693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f1693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f1698a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f1698a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f1698a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f16937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.188 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f16937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.189 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.180+0000 7f16937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.189 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 1 -- 172.21.15.129:0/3506589981 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f169413f5c0 msgr2=0x7f1694135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f169413f5c0 0x7f1694135dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 1 -- 172.21.15.129:0/3506589981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16940a8d70 msgr2=0x7f16941436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16940a8d70 0x7f16941436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 1 monclient: found mon.noname-b 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.182+0000 7f1693fff640 1 -- 172.21.15.129:0/3506589981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688009400 con 0x7f16940a8990 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.185+0000 7f1693fff640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16940a8990 0x7f1694143190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f168400c390 tx=0x7f168400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.185+0000 7f1690ff9640 10 client.?.objecter ms_handle_connect 0x7f16940a8990 2023-12-07T23:46:09.190 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.185+0000 7f1690ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 1 -- 172.21.15.129:0/3506589981 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684005ce0 con 0x7f16940a8990 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: dump: 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.192 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: _finish_auth 0 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 1 -- 172.21.15.129:0/3506589981 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684005e40 con 0x7f16940a8990 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 5 monclient: authenticate success, global_id 4264 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 10 monclient: _renew_subs 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1694144e10 con 0x7f16940a8990 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 1 -- 172.21.15.129:0/3506589981 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16840072f0 con 0x7f16940a8990 2023-12-07T23:46:09.193 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f1690ff9640 10 monclient: dump: 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:epoch 1 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr: 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:09.194 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 10 monclient: _renew_subs 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f16927fc640 4 set_mon_vals no callback set 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1694136310 con 0x7f16940a8990 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.188+0000 7f169a4c9640 1 librados: init done 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.189+0000 7f169a4c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.189+0000 7f169a4c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.189+0000 7f169a4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.195 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.189+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16941d6da0 con 0x7f16940a8990 2023-12-07T23:46:09.210 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.207+0000 7f1690ff9640 1 -- 172.21.15.129:0/3506589981 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f1684007450 con 0x7f16940a8990 2023-12-07T23:46:09.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.208+0000 7f1690ff9640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f168006c7e0 0x7f168006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.208+0000 7f16937fe640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f168006c7e0 0x7f168006eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.208+0000 7f16937fe640 10 monclient: get_auth_request con 0x7f168006c7e0 auth_method 0 2023-12-07T23:46:09.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.208+0000 7f16937fe640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f168006c7e0 0x7f168006eca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1688036000 tx=0x7f168803c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:09.211 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.208+0000 7f1690ff9640 10 client.4264.objecter ms_handle_connect 0x7f168006c7e0 2023-12-07T23:46:09.213 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.210+0000 7f1690ff9640 1 -- 172.21.15.129:0/3506589981 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1684057c90 con 0x7f16940a8990 2023-12-07T23:46:09.213 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.210+0000 7f1690ff9640 10 client.4264.objecter ms_dispatch 0x7f1694000b90 osd_map(17..17 src has 1..17) v4 2023-12-07T23:46:09.213 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.210+0000 7f1690ff9640 3 client.4264.objecter handle_osd_map got epochs [17,17] > 0 2023-12-07T23:46:09.213 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.210+0000 7f1690ff9640 3 client.4264.objecter handle_osd_map decoding full epoch 17 2023-12-07T23:46:09.213 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.210+0000 7f1690ff9640 20 client.4264.objecter dump_active .. 0 homeless 2023-12-07T23:46:09.442 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.439+0000 7f1690ff9640 1 -- 172.21.15.129:0/3506589981 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1684053fd0 con 0x7f16940a8990 2023-12-07T23:46:09.443 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.439+0000 7f1690ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:09.443 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.439+0000 7f1690ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:09.621 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.618+0000 7f169a4c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:09.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.618+0000 7f169a4c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:09.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.618+0000 7f169a4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.622 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.618+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16941d6da0 con 0x7f16940a8990 2023-12-07T23:46:09.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.705+0000 7f1690ff9640 1 -- 172.21.15.129:0/3506589981 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f16941d6da0 con 0x7f16940a8990 2023-12-07T23:46:09.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.705+0000 7f1690ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:09.709 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.705+0000 7f1690ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:09.709 INFO:teuthology.misc.health.smithi129.stdout: 2023-12-07T23:46:09.709 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":17,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:46:06.605925+0000","last_up_change":"2023-12-07T23:46:04.801550+0000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T23:46:02.095929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6825","nonce":771847878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6827","nonce":771847878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6831","nonce":771847878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6829","nonce":771847878}]},"public_addr":"172.21.15.129:6825/771847878","cluster_addr":"172.21.15.129:6827/771847878","heartbeat_back_addr":"172.21.15.129:6831/771847878","heartbeat_front_addr":"172.21.15.129:6829/771847878","state":["exists","up"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6817","nonce":3964479910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6819","nonce":3964479910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6823","nonce":3964479910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6821","nonce":3964479910}]},"public_addr":"172.21.15.129:6817/3964479910","cluster_addr":"172.21.15.129:6819/3964479910","heartbeat_back_addr":"172.21.15.129:6823/3964479910","heartbeat_front_addr":"172.21.15.129:6821/3964479910","state":["exists","up"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6801","nonce":262618579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6803","nonce":262618579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6807","nonce":262618579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6805","nonce":262618579}]},"public_addr":"172.21.15.129:6801/262618579","cluster_addr":"172.21.15.129:6803/262618579","heartbeat_back_addr":"172.21.15.129:6807/262618579","heartbeat_front_addr":"172.21.15.129:6805/262618579","state":["exists","up"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6809","nonce":2348435352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6811","nonce":2348435352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6815","nonce":2348435352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6813","nonce":2348435352}]},"public_addr":"172.21.15.129:6809/2348435352","cluster_addr":"172.21.15.129:6811/2348435352","heartbeat_back_addr":"172.21.15.129:6815/2348435352","heartbeat_front_addr":"172.21.15.129:6813/2348435352","state":["exists","up"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6827","nonce":917696104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6829","nonce":917696104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6833","nonce":917696104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6831","nonce":917696104}]},"public_addr":"172.21.15.184:6827/917696104","cluster_addr":"172.21.15.184:6829/917696104","heartbeat_back_addr":"172.21.15.184:6833/917696104","heartbeat_front_addr":"172.21.15.184:6831/917696104","state":["exists","up"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6819","nonce":4213359653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6821","nonce":4213359653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6825","nonce":4213359653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6823","nonce":4213359653}]},"public_addr":"172.21.15.184:6819/4213359653","cluster_addr":"172.21.15.184:6821/4213359653","heartbeat_back_addr":"172.21.15.184:6825/4213359653","heartbeat_front_addr":"172.21.15.184:6823/4213359653","state":["exists","up"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6807","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6809","nonce":105483051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6813","nonce":105483051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6817","nonce":105483051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6815","nonce":105483051}]},"public_addr":"172.21.15.184:6809/105483051","cluster_addr":"172.21.15.184:6813/105483051","heartbeat_back_addr":"172.21.15.184:6817/105483051","heartbeat_front_addr":"172.21.15.184:6815/105483051","state":["exists","up"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6803","nonce":59760990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6805","nonce":59760990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6811","nonce":59760990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6808","nonce":59760990}]},"public_addr":"172.21.15.184:6803/59760990","cluster_addr":"172.21.15.184:6805/59760990","heartbeat_back_addr":"172.21.15.184:6811/59760990","heartbeat_front_addr":"172.21.15.184:6808/59760990","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.382162+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:00.490549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.542980+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.347947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.274789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.915445+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.493984+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.330992+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:46:09.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 10 librados: watch_flush enter 2023-12-07T23:46:09.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 10 librados: watch_flush exit 2023-12-07T23:46:09.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 20 client.4264.objecter shutdown clearing up homeless session... 2023-12-07T23:46:09.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 10 client.4264.objecter successfully canceled tick 2023-12-07T23:46:09.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f168006c7e0 msgr2=0x7f168006eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f168006c7e0 0x7f168006eca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1688036000 tx=0x7f168803c040 comp rx=0 tx=0).stop 2023-12-07T23:46:09.712 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 10 monclient: shutdown 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16940a8990 msgr2=0x7f1694143190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16940a8990 0x7f1694143190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f168400c390 tx=0x7f168400c850 comp rx=0 tx=0).stop 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 shutdown_connections 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f168006c7e0 0x7f168006eca0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f169413f5c0 0x7f1694135dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16940a8d70 0x7f16941436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 --2- 172.21.15.129:0/3506589981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16940a8990 0x7f1694143190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 >> 172.21.15.129:0/3506589981 conn(0x7f1694066dd0 msgr2=0x7f16940b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 shutdown_connections 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.708+0000 7f169a4c9640 1 -- 172.21.15.129:0/3506589981 wait complete. 2023-12-07T23:46:09.713 INFO:teuthology.misc.health.smithi129.stderr:2023-12-07T23:46:09.709+0000 7f169a4c9640 1 librados: shutdown 2023-12-07T23:46:09.725 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-07T23:46:09.725 INFO:tasks.ceph:Creating RBD pool 2023-12-07T23:46:09.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-07T23:46:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.825+0000 7f7b6f0d7640 10 monclient: get_monmap_and_config 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.825+0000 7f7b6f0d7640 10 monclient: build_initial_monmap 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.825+0000 7f7b6f0d7640 10 monclient: monmap: 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:09.826984+0000 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:09.826984+0000 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:09.830 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.826+0000 7f7b6f0d7640 1 Processor -- start 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.826+0000 7f7b6f0d7640 1 -- start start 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.826+0000 7f7b6f0d7640 10 monclient: init 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7b680b2c50 0x7f7b68071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient(hunting): picked mon.noname-a con 0x7f7b680b2c50 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b680afaf0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7b6813ef70 0x7f7b6813f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6813ef70 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b680701d0 con 0x7f7b680afaf0 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b68071cb0 con 0x7f7b680b2c50 2023-12-07T23:46:09.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6806d4f0 con 0x7f7b6813ef70 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6f0d7640 10 monclient(hunting): authenticate will time out at 1370.015309s 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.827+0000 7f7b6ce73640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7b680b2c50 0x7f7b68071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.828+0000 7f7b67fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7b680afaf0 0x7f7b6806cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55580/0 (socket says 172.21.15.129:55580) 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.828+0000 7f7b67fff640 1 -- 172.21.15.129:0/3638081837 learned_addr learned my addr 172.21.15.129:0/3638081837 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.828+0000 7f7b6ce73640 10 monclient(hunting): get_auth_request con 0x7f7b680b2c50 auth_method 0 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.828+0000 7f7b6ce73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.828+0000 7f7b6ce73640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.828+0000 7f7b6ce73640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.837 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.829+0000 7f7b6ce73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.829+0000 7f7b6ce73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.829+0000 7f7b6ce73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.829+0000 7f7b6d674640 1 --2- 172.21.15.129:0/3638081837 >> v2:172.21.15.184:3300/0 conn(0x7f7b6813ef70 0x7f7b6813f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.830+0000 7f7b6d674640 10 monclient(hunting): get_auth_request con 0x7f7b6813ef70 auth_method 0 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.830+0000 7f7b6d674640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.830+0000 7f7b6d674640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.830+0000 7f7b6d674640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.832+0000 7f7b6d674640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.832+0000 7f7b6d674640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.832+0000 7f7b6d674640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 1 -- 172.21.15.129:0/3638081837 >> v2:172.21.15.129:3300/0 conn(0x7f7b680b2c50 msgr2=0x7f7b68071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 1 --2- 172.21.15.129:0/3638081837 >> v2:172.21.15.129:3300/0 conn(0x7f7b680b2c50 0x7f7b68071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 1 -- 172.21.15.129:0/3638081837 >> v1:172.21.15.129:6789/0 conn(0x7f7b680afaf0 legacy=0x7f7b6806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 10 monclient: _finish_hunting 0 2023-12-07T23:46:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 1 monclient: found mon.noname-c 2023-12-07T23:46:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.833+0000 7f7b6d674640 1 -- 172.21.15.129:0/3638081837 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68071070 con 0x7f7b6813ef70 2023-12-07T23:46:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.835+0000 7f7b6d674640 1 --2- 172.21.15.129:0/3638081837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813ef70 0x7f7b6813f3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b6806be10 tx=0x7f7b5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3dfd122e6b9d0c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 1 -- 172.21.15.129:0/3638081837 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58031830 con 0x7f7b6813ef70 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: dump: 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.843 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 1 -- 172.21.15.129:0/3638081837 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b58031990 con 0x7f7b6813ef70 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b6f0d7640 5 monclient: authenticate success, global_id 4267 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b6f0d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b6f0d7640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b6f0d7640 4 set_mon_vals no callback set 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 1 -- 172.21.15.129:0/3638081837 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58031d20 con 0x7f7b6813ef70 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.839+0000 7f7b677fe640 10 monclient: dump: 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 10 monclient: shutdown 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/3638081837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813ef70 msgr2=0x7f7b6813f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/3638081837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813ef70 0x7f7b6813f3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b6806be10 tx=0x7f7b5802b8d0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/3638081837 shutdown_connections 2023-12-07T23:46:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/3638081837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813ef70 0x7f7b6813f3c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/3638081837 >> v2:172.21.15.129:3300/0 conn(0x7f7b680b2c50 0x7f7b68071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/3638081837 >> 172.21.15.129:0/3638081837 conn(0x7f7b680b5da0 msgr2=0x7f7b680b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/3638081837 shutdown_connections 2023-12-07T23:46:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/3638081837 wait complete. 2023-12-07T23:46:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 10 monclient: build_initial_monmap 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.840+0000 7f7b6f0d7640 10 monclient: monmap: 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 1 librados: starting msgr at 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 1 librados: starting objecter 2023-12-07T23:46:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 1 Processor -- start 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 1 -- start start 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 1 librados: setting wanted keys 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 1 librados: calling monclient init 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 10 monclient: init 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.841+0000 7f7b6f0d7640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b680afaf0 0x7f7b680b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): picked mon.noname-b con 0x7f7b680afaf0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680b2c50 0x7f7b6813fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): picked mon.noname-c con 0x7f7b680b2c50 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6813ef70 0x7f7b68142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6813ef70 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b681420f0 con 0x7f7b6813ef70 2023-12-07T23:46:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b68136360 con 0x7f7b680afaf0 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b68136590 con 0x7f7b680b2c50 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6f0d7640 10 monclient(hunting): authenticate will time out at 1370.030028s 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b67fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680b2c50 0x7f7b6813fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6ce73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b680afaf0 0x7f7b680b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6ce73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b680afaf0 0x7f7b680b4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:51218/0 (socket says 172.21.15.129:51218) 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6ce73640 1 -- 172.21.15.129:0/754770483 learned_addr learned my addr 172.21.15.129:0/754770483 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6ce73640 10 monclient(hunting): get_auth_request con 0x7f7b680afaf0 auth_method 0 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6ce73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6ce73640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b6ce73640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b67fff640 10 monclient(hunting): get_auth_request con 0x7f7b680b2c50 auth_method 0 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b67fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.842+0000 7f7b67fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.843+0000 7f7b6ce73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.843+0000 7f7b6ce73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.843+0000 7f7b6ce73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.843+0000 7f7b67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.843+0000 7f7b67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.843+0000 7f7b67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.845+0000 7f7b6ce73640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.845+0000 7f7b6d674640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6813ef70 0x7f7b68142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.845+0000 7f7b6ce73640 1 -- 172.21.15.129:0/754770483 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680b2c50 msgr2=0x7f7b6813fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.845+0000 7f7b6ce73640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680b2c50 0x7f7b6813fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.845+0000 7f7b6ce73640 1 -- 172.21.15.129:0/754770483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6813ef70 msgr2=0x7f7b68142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.845+0000 7f7b6ce73640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6813ef70 0x7f7b68142540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.846+0000 7f7b6ce73640 10 monclient: _finish_hunting 0 2023-12-07T23:46:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.846+0000 7f7b6ce73640 1 monclient: found mon.noname-b 2023-12-07T23:46:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.846+0000 7f7b6ce73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.846+0000 7f7b6ce73640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.846+0000 7f7b6ce73640 1 -- 172.21.15.129:0/754770483 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58009400 con 0x7f7b680afaf0 2023-12-07T23:46:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.848+0000 7f7b6ce73640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b680afaf0 0x7f7b680b4a80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000efc0 tx=0x7f7b6000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.848+0000 7f7b657fa640 10 client.?.objecter ms_handle_connect 0x7f7b680afaf0 2023-12-07T23:46:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.848+0000 7f7b657fa640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60014480 con 0x7f7b680afaf0 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: dump: 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: _finish_auth 0 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: _check_auth_tickets 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 5 monclient: authenticate success, global_id 4270 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6000f040 con 0x7f7b680afaf0 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60014800 con 0x7f7b680afaf0 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 10 monclient: _renew_subs 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b68136870 con 0x7f7b680afaf0 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b66ffd640 4 set_mon_vals no callback set 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b657fa640 10 monclient: dump: 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 10 monclient: _renew_subs 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b68142a80 con 0x7f7b680afaf0 2023-12-07T23:46:09.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.851+0000 7f7b6f0d7640 1 librados: init done 2023-12-07T23:46:09.858 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.852+0000 7f7b6f0d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:09.858 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.852+0000 7f7b6f0d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:09.858 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.852+0000 7f7b6f0d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:09.858 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.852+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6804e840 con 0x7f7b680afaf0 2023-12-07T23:46:09.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.870+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f7b60014960 con 0x7f7b680afaf0 2023-12-07T23:46:09.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.870+0000 7f7b657fa640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b5406c7e0 0x7f7b5406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:09.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.870+0000 7f7b67fff640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b5406c7e0 0x7f7b5406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:09.873 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.870+0000 7f7b67fff640 10 monclient: get_auth_request con 0x7f7b5406c7e0 auth_method 0 2023-12-07T23:46:09.874 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.871+0000 7f7b657fa640 10 client.4270.objecter ms_handle_connect 0x7f7b5406c7e0 2023-12-07T23:46:09.874 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.871+0000 7f7b67fff640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b5406c7e0 0x7f7b5406eca0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7b6814c6d0 tx=0x7f7b5c007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:09.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.873+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b60059dc0 con 0x7f7b680afaf0 2023-12-07T23:46:09.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.873+0000 7f7b657fa640 10 client.4270.objecter ms_dispatch 0x7f7b68000b90 osd_map(17..17 src has 1..17) v4 2023-12-07T23:46:09.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.873+0000 7f7b657fa640 3 client.4270.objecter handle_osd_map got epochs [17,17] > 0 2023-12-07T23:46:09.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.873+0000 7f7b657fa640 3 client.4270.objecter handle_osd_map decoding full epoch 17 2023-12-07T23:46:09.876 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:09.873+0000 7f7b657fa640 20 client.4270.objecter dump_active .. 0 homeless 2023-12-07T23:46:10.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.115+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b60055a00 con 0x7f7b680afaf0 2023-12-07T23:46:10.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.115+0000 7f7b657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:10.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.115+0000 7f7b657fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:10.319 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.316+0000 7f7b6f0d7640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-07T23:46:10.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.316+0000 7f7b6f0d7640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-07T23:46:10.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.316+0000 7f7b6f0d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:10.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.316+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f7b6806aeb0 con 0x7f7b680afaf0 2023-12-07T23:46:10.512 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:10.509+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7b60055dd0 con 0x7f7b680afaf0 2023-12-07T23:46:11.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.344+0000 7f7b657fa640 1 -- 172.21.15.129:0/754770483 <== mon.1 v2:172.21.15.184:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v18) v1 ==== 109+0+0 (secure 0 0 0) 0x7f7b60076d20 con 0x7f7b680afaf0 2023-12-07T23:46:11.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.344+0000 7f7b657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-07T23:46:11.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.344+0000 7f7b657fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-12-07T23:46:11.347 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'rbd' created 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 10 librados: watch_flush enter 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 10 librados: watch_flush exit 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 20 client.4270.objecter shutdown clearing up homeless session... 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 10 client.4270.objecter successfully canceled tick 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b5406c7e0 msgr2=0x7f7b5406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b5406c7e0 0x7f7b5406eca0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7b6814c6d0 tx=0x7f7b5c007e80 comp rx=0 tx=0).stop 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 10 monclient: shutdown 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b680afaf0 msgr2=0x7f7b680b4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b680afaf0 0x7f7b680b4a80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000efc0 tx=0x7f7b6000c490 comp rx=0 tx=0).stop 2023-12-07T23:46:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 shutdown_connections 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b5406c7e0 0x7f7b5406eca0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6813ef70 0x7f7b68142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680b2c50 0x7f7b6813fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 --2- 172.21.15.129:0/754770483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b680afaf0 0x7f7b680b4a80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 >> 172.21.15.129:0/754770483 conn(0x7f7b680b5da0 msgr2=0x7f7b68134090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 shutdown_connections 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 -- 172.21.15.129:0/754770483 wait complete. 2023-12-07T23:46:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.346+0000 7f7b6f0d7640 1 librados: shutdown 2023-12-07T23:46:11.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-07T23:46:11.472 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.469+0000 7f4a3c84f640 10 monclient: get_monmap_and_config 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.469+0000 7f4a3c84f640 10 monclient: build_initial_monmap 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.469+0000 7f4a3c84f640 10 monclient: monmap: 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:11.470943+0000 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:11.470943+0000 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:11.474 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:11.474 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:11.475 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.470+0000 7f4a3c84f640 1 Processor -- start 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.470+0000 7f4a3c84f640 1 -- start start 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.470+0000 7f4a3c84f640 10 monclient: init 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a380a8910 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8cf0 0x7f4a38142200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient(hunting): picked mon.noname-b con 0x7f4a380a8cf0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a38142740 0x7f4a3814c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient(hunting): picked mon.noname-a con 0x7f4a38142740 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a380b20e0 con 0x7f4a380a8910 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a380b18f0 con 0x7f4a38142740 2023-12-07T23:46:11.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380aab60 con 0x7f4a380a8cf0 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.471+0000 7f4a3c84f640 10 monclient(hunting): authenticate will time out at 1371.659499s 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a35d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8cf0 0x7f4a38142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a36d9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a38142740 0x7f4a3814c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a35d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8cf0 0x7f4a38142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:48932/0 (socket says 172.21.15.129:48932) 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a35d9b640 1 -- 172.21.15.129:0/1588768502 learned_addr learned my addr 172.21.15.129:0/1588768502 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a35d9b640 10 monclient(hunting): get_auth_request con 0x7f4a380a8cf0 auth_method 0 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a35d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a35d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a36d9d640 10 monclient(hunting): get_auth_request con 0x7f4a38142740 auth_method 0 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:11.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a36d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:11.478 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.472+0000 7f4a36d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:11.478 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.473+0000 7f4a35d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:11.478 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.473+0000 7f4a35d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:11.478 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.473+0000 7f4a35d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.473+0000 7f4a36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.473+0000 7f4a36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.473+0000 7f4a36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 1 -- 172.21.15.129:0/1588768502 >> v2:172.21.15.129:3300/0 conn(0x7f4a38142740 msgr2=0x7f4a3814c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 1 --2- 172.21.15.129:0/1588768502 >> v2:172.21.15.129:3300/0 conn(0x7f4a38142740 0x7f4a3814c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 1 -- 172.21.15.129:0/1588768502 >> v1:172.21.15.129:6789/0 conn(0x7f4a380a8910 legacy=0x7f4a3813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 10 monclient: _finish_hunting 0 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 1 monclient: found mon.noname-b 2023-12-07T23:46:11.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:11.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:11.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.474+0000 7f4a35d9b640 1 -- 172.21.15.129:0/1588768502 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a380b1580 con 0x7f4a380a8cf0 2023-12-07T23:46:11.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.476+0000 7f4a35d9b640 1 --2- 172.21.15.129:0/1588768502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8cf0 0x7f4a38142200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009870 tx=0x7f4a2c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f92628980bd155a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:11.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 1 -- 172.21.15.129:0/1588768502 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c031640 con 0x7f4a380a8cf0 2023-12-07T23:46:11.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: dump: 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: _finish_auth 0 2023-12-07T23:46:11.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: _check_auth_tickets 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 1 -- 172.21.15.129:0/1588768502 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c0317a0 con 0x7f4a380a8cf0 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3c84f640 5 monclient: authenticate success, global_id 4220 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3c84f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3c84f640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 1 -- 172.21.15.129:0/1588768502 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c031b30 con 0x7f4a380a8cf0 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3c84f640 4 set_mon_vals no callback set 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.480+0000 7f4a3559a640 10 monclient: dump: 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.486 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:11.486 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:11.486 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:11.486 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:11.486 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:11.486 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:11.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 10 monclient: shutdown 2023-12-07T23:46:11.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:11.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 -- 172.21.15.129:0/1588768502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8cf0 msgr2=0x7f4a38142200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:11.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/1588768502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8cf0 0x7f4a38142200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009870 tx=0x7f4a2c02b8f0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 -- 172.21.15.129:0/1588768502 shutdown_connections 2023-12-07T23:46:11.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/1588768502 >> v2:172.21.15.129:3300/0 conn(0x7f4a38142740 0x7f4a3814c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/1588768502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8cf0 0x7f4a38142200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 -- 172.21.15.129:0/1588768502 >> 172.21.15.129:0/1588768502 conn(0x7f4a38066c40 msgr2=0x7f4a380b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 -- 172.21.15.129:0/1588768502 shutdown_connections 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.481+0000 7f4a3c84f640 1 -- 172.21.15.129:0/1588768502 wait complete. 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 10 monclient: build_initial_monmap 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 10 monclient: monmap: 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:11.490 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 1 librados: starting msgr at 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 1 librados: starting objecter 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 1 Processor -- start 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 1 -- start start 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 1 librados: setting wanted keys 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 1 librados: calling monclient init 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.482+0000 7f4a3c84f640 10 monclient: init 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8910 0x7f4a3813e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): picked mon.noname-c con 0x7f4a380a8910 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a380a8cf0 0x7f4a3813b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): picked mon.noname-b con 0x7f4a380a8cf0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a38142740 0x7f4a38133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): picked mon.noname-a con 0x7f4a38142740 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3813c2d0 con 0x7f4a38142740 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3813bc70 con 0x7f4a380a8cf0 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3813c760 con 0x7f4a380a8910 2023-12-07T23:46:11.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.483+0000 7f4a3c84f640 10 monclient(hunting): authenticate will time out at 1371.671464s 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8910 0x7f4a3813e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a36d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a38142740 0x7f4a38133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a35d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a380a8cf0 0x7f4a3813b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8910 0x7f4a3813e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:48944/0 (socket says 172.21.15.129:48944) 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 1 -- 172.21.15.129:0/2724202810 learned_addr learned my addr 172.21.15.129:0/2724202810 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 10 monclient(hunting): get_auth_request con 0x7f4a380a8910 auth_method 0 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a36d9d640 10 monclient(hunting): get_auth_request con 0x7f4a38142740 auth_method 0 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a36d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a36d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a35d9b640 10 monclient(hunting): get_auth_request con 0x7f4a380a8cf0 auth_method 0 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a35d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.484+0000 7f4a35d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.485+0000 7f4a36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.485+0000 7f4a36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.485+0000 7f4a36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.485+0000 7f4a35d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.485+0000 7f4a35d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.485+0000 7f4a35d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2023-12-07T23:46:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 1 -- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a380a8cf0 msgr2=0x7f4a3813b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a380a8cf0 0x7f4a3813b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 1 -- 172.21.15.129:0/2724202810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a38142740 msgr2=0x7f4a38133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a38142740 0x7f4a38133d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 10 monclient: _finish_hunting 0 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 1 monclient: found mon.noname-c 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.487+0000 7f4a3659c640 1 -- 172.21.15.129:0/2724202810 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0094d0 con 0x7f4a380a8910 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.490+0000 7f4a3659c640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8910 0x7f4a3813e750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c430 tx=0x7f4a2800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.490+0000 7f4a26ffd640 10 client.?.objecter ms_handle_connect 0x7f4a380a8910 2023-12-07T23:46:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.490+0000 7f4a26ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:11.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 1 -- 172.21.15.129:0/2724202810 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280052d0 con 0x7f4a380a8910 2023-12-07T23:46:11.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:11.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:11.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: dump: 2023-12-07T23:46:11.497 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 1 -- 172.21.15.129:0/2724202810 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a28005450 con 0x7f4a380a8910 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 1 -- 172.21.15.129:0/2724202810 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28007260 con 0x7f4a380a8910 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 5 monclient: authenticate success, global_id 4223 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 10 monclient: _renew_subs 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a381446d0 con 0x7f4a380a8910 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a34d99640 4 set_mon_vals no callback set 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a26ffd640 10 monclient: dump: 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 10 monclient: _renew_subs 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a38134240 con 0x7f4a380a8910 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.494+0000 7f4a3c84f640 1 librados: init done 2023-12-07T23:46:11.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.495+0000 7f4a3c84f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:11.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.495+0000 7f4a3c84f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:11.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.495+0000 7f4a3c84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:11.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.495+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a38067ba0 con 0x7f4a380a8910 2023-12-07T23:46:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.513+0000 7f4a26ffd640 1 -- 172.21.15.129:0/2724202810 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4a280073c0 con 0x7f4a380a8910 2023-12-07T23:46:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.513+0000 7f4a26ffd640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a1806c820 0x7f4a1806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.514+0000 7f4a35d9b640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a1806c820 0x7f4a1806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.514+0000 7f4a35d9b640 10 monclient: get_auth_request con 0x7f4a1806c820 auth_method 0 2023-12-07T23:46:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.514+0000 7f4a26ffd640 10 client.4223.objecter ms_handle_connect 0x7f4a1806c820 2023-12-07T23:46:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.514+0000 7f4a35d9b640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a1806c820 0x7f4a1806ece0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c036000 tx=0x7f4a2c03c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.516+0000 7f4a26ffd640 1 -- 172.21.15.129:0/2724202810 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f4a28058040 con 0x7f4a380a8910 2023-12-07T23:46:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.516+0000 7f4a26ffd640 10 client.4223.objecter ms_dispatch 0x7f4a38000b90 osd_map(18..18 src has 1..18) v4 2023-12-07T23:46:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.516+0000 7f4a26ffd640 3 client.4223.objecter handle_osd_map got epochs [18,18] > 0 2023-12-07T23:46:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.516+0000 7f4a26ffd640 3 client.4223.objecter handle_osd_map decoding full epoch 18 2023-12-07T23:46:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.516+0000 7f4a26ffd640 20 client.4223.objecter dump_active .. 0 homeless 2023-12-07T23:46:11.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.761+0000 7f4a26ffd640 1 -- 172.21.15.129:0/2724202810 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a28054380 con 0x7f4a380a8910 2023-12-07T23:46:11.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.761+0000 7f4a26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:11.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.761+0000 7f4a26ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.966+0000 7f4a3c84f640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-07T23:46:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.966+0000 7f4a3c84f640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-07T23:46:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.966+0000 7f4a3c84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:11.966+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4a3813bdb0 con 0x7f4a380a8910 2023-12-07T23:46:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.966+0000 7f4a26ffd640 1 -- 172.21.15.129:0/2724202810 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v19) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4a3813bdb0 con 0x7f4a380a8910 2023-12-07T23:46:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.966+0000 7f4a26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-07T23:46:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.966+0000 7f4a26ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-12-07T23:46:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-07T23:46:12.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 10 librados: watch_flush enter 2023-12-07T23:46:12.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 10 librados: watch_flush exit 2023-12-07T23:46:12.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 20 client.4223.objecter shutdown clearing up homeless session... 2023-12-07T23:46:12.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 10 client.4223.objecter successfully canceled tick 2023-12-07T23:46:12.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a1806c820 msgr2=0x7f4a1806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:12.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a1806c820 0x7f4a1806ece0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c036000 tx=0x7f4a2c03c040 comp rx=0 tx=0).stop 2023-12-07T23:46:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 10 monclient: shutdown 2023-12-07T23:46:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8910 msgr2=0x7f4a3813e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8910 0x7f4a3813e750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c430 tx=0x7f4a2800c8f0 comp rx=0 tx=0).stop 2023-12-07T23:46:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 shutdown_connections 2023-12-07T23:46:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a1806c820 0x7f4a1806ece0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a38142740 0x7f4a38133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a380a8cf0 0x7f4a3813b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 --2- 172.21.15.129:0/2724202810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a380a8910 0x7f4a3813e750 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.969+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 >> 172.21.15.129:0/2724202810 conn(0x7f4a38066c40 msgr2=0x7f4a3813f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.970+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 shutdown_connections 2023-12-07T23:46:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.970+0000 7f4a3c84f640 1 -- 172.21.15.129:0/2724202810 wait complete. 2023-12-07T23:46:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:12.970+0000 7f4a3c84f640 1 librados: shutdown 2023-12-07T23:46:12.984 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-07T23:46:12.985 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-07T23:46:12.985 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-07T23:46:12.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-07T23:46:12.986 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-07T23:46:13.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.102+0000 7fb0e34c2640 10 monclient: get_monmap_and_config 2023-12-07T23:46:13.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.102+0000 7fb0e34c2640 10 monclient: build_initial_monmap 2023-12-07T23:46:13.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.102+0000 7fb0e34c2640 10 monclient: monmap: 2023-12-07T23:46:13.105 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:13.105 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:13.105 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:13.103489+0000 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:13.103489+0000 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:13.106 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.102+0000 7fb0e34c2640 1 Processor -- start 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 1 -- start start 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient: init 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb0dc066280 0x7fb0dc0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient(hunting): picked mon.noname-c con 0x7fb0dc066280 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0dc066660 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc145970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient(hunting): picked mon.noname-b con 0x7fb0dc145520 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:13.108 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0dc0af710 con 0x7fb0dc066660 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0dc0aef40 con 0x7fb0dc066280 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc0aeb80 con 0x7fb0dc145520 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.103+0000 7fb0e34c2640 10 monclient(hunting): authenticate will time out at 1373.291790s 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.104+0000 7fb0e125e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb0dc066280 0x7fb0dc0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.104+0000 7fb0e1a5f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e125e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb0dc066280 0x7fb0dc0afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40406/0 (socket says 172.21.15.129:40406) 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e125e640 1 -- 172.21.15.129:0/3946578776 learned_addr learned my addr 172.21.15.129:0/3946578776 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e125e640 10 monclient(hunting): get_auth_request con 0x7fb0dc066280 auth_method 0 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e125e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e125e640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:13.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e125e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:13.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e1a5f640 10 monclient(hunting): get_auth_request con 0x7fb0dc145520 auth_method 0 2023-12-07T23:46:13.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e1a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:13.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e1a5f640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:13.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e1a5f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:13.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e1a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:13.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e1a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:13.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.105+0000 7fb0e1a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.106+0000 7fb0e125e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.106+0000 7fb0e125e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.106+0000 7fb0e125e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 1 -- 172.21.15.129:0/3946578776 >> v2:172.21.15.184:3300/0 conn(0x7fb0dc066280 msgr2=0x7fb0dc0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 1 --2- 172.21.15.129:0/3946578776 >> v2:172.21.15.184:3300/0 conn(0x7fb0dc066280 0x7fb0dc0afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 1 -- 172.21.15.129:0/3946578776 >> v1:172.21.15.184:6789/0 conn(0x7fb0dc066660 legacy=0x7fb0dc13baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 10 monclient: _finish_hunting 0 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 1 monclient: found mon.noname-b 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:13.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.108+0000 7fb0e1a5f640 1 -- 172.21.15.129:0/3946578776 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc0aecc0 con 0x7fb0dc145520 2023-12-07T23:46:13.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.111+0000 7fb0e1a5f640 1 --2- 172.21.15.129:0/3946578776 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc145970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009d90 tx=0x7fb0d402bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c0f5e4259dcc41 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:13.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 1 -- 172.21.15.129:0/3946578776 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d402caa0 con 0x7fb0dc145520 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: dump: 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: _finish_auth 0 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: _check_auth_tickets 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 1 -- 172.21.15.129:0/3946578776 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d402cc00 con 0x7fb0dc145520 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 1 -- 172.21.15.129:0/3946578776 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4034440 con 0x7fb0dc145520 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0d3fff640 10 monclient: dump: 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:13.336 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:13.337 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0e34c2640 5 monclient: authenticate success, global_id 4229 2023-12-07T23:46:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0e34c2640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.331+0000 7fb0e34c2640 4 set_mon_vals no callback set 2023-12-07T23:46:13.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 10 monclient: shutdown 2023-12-07T23:46:13.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:13.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3946578776 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 msgr2=0x7fb0dc145970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3946578776 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc145970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009d90 tx=0x7fb0d402bcf0 comp rx=0 tx=0).stop 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3946578776 shutdown_connections 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3946578776 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc145970 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3946578776 >> v2:172.21.15.184:3300/0 conn(0x7fb0dc066280 0x7fb0dc0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.332+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3946578776 >> 172.21.15.129:0/3946578776 conn(0x7fb0dc0b3cc0 msgr2=0x7fb0dc0b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.333+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3946578776 shutdown_connections 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.333+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3946578776 wait complete. 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 10 monclient: build_initial_monmap 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 10 monclient: monmap: 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:13.340 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 1 librados: starting msgr at 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 1 librados: starting objecter 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 1 Processor -- start 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 1 -- start start 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 1 librados: setting wanted keys 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 1 librados: calling monclient init 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.334+0000 7fb0e34c2640 10 monclient: init 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:46:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc066280 0x7fb0dc13e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): picked mon.noname-b con 0x7fb0dc066280 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): picked mon.noname-c con 0x7fb0dc145520 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc141230 0x7fb0dc13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): picked mon.noname-a con 0x7fb0dc141230 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc1416e0 con 0x7fb0dc141230 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc0a8640 con 0x7fb0dc066280 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc0a88b0 con 0x7fb0dc145520 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.335+0000 7fb0e34c2640 10 monclient(hunting): authenticate will time out at 1373.523753s 2023-12-07T23:46:13.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e0a5d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e125e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc066280 0x7fb0dc13e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e0a5d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55434/0 (socket says 172.21.15.129:55434) 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e0a5d640 1 -- 172.21.15.129:0/3259611129 learned_addr learned my addr 172.21.15.129:0/3259611129 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e0a5d640 10 monclient(hunting): get_auth_request con 0x7fb0dc145520 auth_method 0 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e0a5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e0a5d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e0a5d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e125e640 10 monclient(hunting): get_auth_request con 0x7fb0dc066280 auth_method 0 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e125e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e125e640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.336+0000 7fb0e125e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.337+0000 7fb0e0a5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.337+0000 7fb0e0a5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.337+0000 7fb0e0a5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.337+0000 7fb0e125e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.337+0000 7fb0e125e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.337+0000 7fb0e125e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 1 -- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc066280 msgr2=0x7fb0dc13e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc066280 0x7fb0dc13e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 1 -- 172.21.15.129:0/3259611129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc141230 msgr2=0x7fb0dc13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc141230 0x7fb0dc13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 10 monclient: _finish_hunting 0 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 1 monclient: found mon.noname-c 2023-12-07T23:46:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.340+0000 7fb0e0a5d640 1 -- 172.21.15.129:0/3259611129 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d4009a70 con 0x7fb0dc145520 2023-12-07T23:46:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.344+0000 7fb0e0a5d640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc142350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00c430 tx=0x7fb0cc00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.344+0000 7fb0d1ffb640 10 client.?.objecter ms_handle_connect 0x7fb0dc145520 2023-12-07T23:46:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.344+0000 7fb0d1ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:13.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 1 -- 172.21.15.129:0/3259611129 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc011790 con 0x7fb0dc145520 2023-12-07T23:46:13.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:13.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:13.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: dump: 2023-12-07T23:46:13.354 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 1 -- 172.21.15.129:0/3259611129 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc0118f0 con 0x7fb0dc145520 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 1 -- 172.21.15.129:0/3259611129 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc011ca0 con 0x7fb0dc145520 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d1ffb640 10 monclient: dump: 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:13.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0d37fe640 4 set_mon_vals no callback set 2023-12-07T23:46:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0e34c2640 5 monclient: authenticate success, global_id 4232 2023-12-07T23:46:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0e34c2640 10 monclient: _renew_subs 2023-12-07T23:46:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0e34c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.351+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0dc0a8b90 con 0x7fb0dc145520 2023-12-07T23:46:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.352+0000 7fb0e34c2640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.352+0000 7fb0e34c2640 10 monclient: _renew_subs 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.352+0000 7fb0e34c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.352+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0dc1401e0 con 0x7fb0dc145520 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.352+0000 7fb0e34c2640 1 librados: init done 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.353+0000 7fb0e34c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.353+0000 7fb0e34c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.353+0000 7fb0e34c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.353+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0dc142070 con 0x7fb0dc145520 2023-12-07T23:46:13.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.381+0000 7fb0d1ffb640 1 -- 172.21.15.129:0/3259611129 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb0cc01c460 con 0x7fb0dc145520 2023-12-07T23:46:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.382+0000 7fb0d1ffb640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb0c006c820 0x7fb0c006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.382+0000 7fb0e125e640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb0c006c820 0x7fb0c006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.383+0000 7fb0e125e640 10 monclient: get_auth_request con 0x7fb0c006c820 auth_method 0 2023-12-07T23:46:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.383+0000 7fb0d1ffb640 10 client.4232.objecter ms_handle_connect 0x7fb0c006c820 2023-12-07T23:46:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.383+0000 7fb0e125e640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb0c006c820 0x7fb0c006ece0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009b30 tx=0x7fb0d8009240 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.387+0000 7fb0d1ffb640 1 -- 172.21.15.129:0/3259611129 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb0cc059280 con 0x7fb0dc145520 2023-12-07T23:46:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.387+0000 7fb0d1ffb640 10 client.4232.objecter ms_dispatch 0x7fb0dc000b90 osd_map(19..19 src has 1..19) v4 2023-12-07T23:46:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.387+0000 7fb0d1ffb640 3 client.4232.objecter handle_osd_map got epochs [19,19] > 0 2023-12-07T23:46:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.387+0000 7fb0d1ffb640 3 client.4232.objecter handle_osd_map decoding full epoch 19 2023-12-07T23:46:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.387+0000 7fb0d1ffb640 20 client.4232.objecter dump_active .. 0 homeless 2023-12-07T23:46:13.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.750+0000 7fb0d1ffb640 1 -- 172.21.15.129:0/3259611129 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0cc054ec0 con 0x7fb0dc145520 2023-12-07T23:46:13.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.750+0000 7fb0d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:13.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.750+0000 7fb0d1ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:13.906 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:10.379 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:13.963 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:10.436 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.002 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.999+0000 7fb0e34c2640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-12-07T23:46:14.002 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.999+0000 7fb0e34c2640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-07T23:46:14.003 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.999+0000 7fb0e34c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:14.003 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:13.999+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb0dc142070 con 0x7fb0dc145520 2023-12-07T23:46:14.004 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:10.477 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.008 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:10.454 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.062 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:10.505 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.175+0000 7fb0d1ffb640 1 -- 172.21.15.129:0/3259611129 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171039 (secure 0 0 0) 0x7fb0dc142070 con 0x7fb0dc145520 2023-12-07T23:46:14.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.175+0000 7fb0d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-07T23:46:14.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.175+0000 7fb0d1ffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:14.181 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:46:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 10 librados: watch_flush enter 2023-12-07T23:46:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 10 librados: watch_flush exit 2023-12-07T23:46:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 20 client.4232.objecter shutdown clearing up homeless session... 2023-12-07T23:46:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 10 client.4232.objecter successfully canceled tick 2023-12-07T23:46:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb0c006c820 msgr2=0x7fb0c006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb0c006c820 0x7fb0c006ece0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009b30 tx=0x7fb0d8009240 comp rx=0 tx=0).stop 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 10 monclient: shutdown 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 msgr2=0x7fb0dc142350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc142350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00c430 tx=0x7fb0cc00c8f0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 shutdown_connections 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb0c006c820 0x7fb0c006ece0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc141230 0x7fb0dc13fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb0dc145520 0x7fb0dc142350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 --2- 172.21.15.129:0/3259611129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc066280 0x7fb0dc13e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 >> 172.21.15.129:0/3259611129 conn(0x7fb0dc0b3cc0 msgr2=0x7fb0dc144430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:14.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 shutdown_connections 2023-12-07T23:46:14.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 -- 172.21.15.129:0/3259611129 wait complete. 2023-12-07T23:46:14.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.178+0000 7fb0e34c2640 1 librados: shutdown 2023-12-07T23:46:14.192 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":5,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":2561365197},{"type":"v1","addr":"172.21.15.184:6801","nonce":2561365197}]},"active_addr":"172.21.15.184:6801/2561365197","active_change":"2023-12-07T23:45:21.269943+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4111,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":4255747792}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1932824546}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1337308596}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1580474494}]}]} 2023-12-07T23:46:14.196 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-07T23:46:14.196 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-07T23:46:14.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T23:46:14.213 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:10.686 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.270 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:10.742 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.294 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:10.739 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.299 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.296+0000 7f50ed0c9640 10 monclient: get_monmap_and_config 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.296+0000 7f50ed0c9640 10 monclient: build_initial_monmap 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.296+0000 7f50ed0c9640 10 monclient: monmap: 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:14.297839+0000 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:14.297839+0000 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:14.300 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:14.301 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:14.301 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:14.301 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.297+0000 7f50ed0c9640 1 Processor -- start 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.297+0000 7f50ed0c9640 1 -- start start 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.297+0000 7f50ed0c9640 10 monclient: init 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50e80a8970 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e81421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient(hunting): picked mon.noname-b con 0x7f50e80a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f50e813f580 0x7f50e81426e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient(hunting): picked mon.noname-a con 0x7f50e813f580 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:14.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80aa950 con 0x7f50e80a8970 2023-12-07T23:46:14.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:14.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e80a90e0 con 0x7f50e813f580 2023-12-07T23:46:14.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:14.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e80aabc0 con 0x7f50e80a9bd0 2023-12-07T23:46:14.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:14.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.298+0000 7f50ed0c9640 10 monclient(hunting): authenticate will time out at 1374.486395s 2023-12-07T23:46:14.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.320+0000 7f50e759e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f50e813f580 0x7f50e81426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:14.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.322+0000 7f50e759e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f50e813f580 0x7f50e81426e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42724/0 (socket says 172.21.15.129:42724) 2023-12-07T23:46:14.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.322+0000 7f50e759e640 1 -- 172.21.15.129:0/4028031532 learned_addr learned my addr 172.21.15.129:0/4028031532 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:14.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.322+0000 7f50e759e640 10 monclient(hunting): get_auth_request con 0x7f50e813f580 auth_method 0 2023-12-07T23:46:14.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.322+0000 7f50e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:14.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.322+0000 7f50e759e640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:14.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.322+0000 7f50e759e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:14.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.324+0000 7f50e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:14.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.324+0000 7f50e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:14.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.324+0000 7f50e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:14.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2023-12-07T23:46:14.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 1 -- 172.21.15.129:0/4028031532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 msgr2=0x7f50e81421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:14.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 1 --2- 172.21.15.129:0/4028031532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e81421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 1 -- 172.21.15.129:0/4028031532 >> v1:172.21.15.184:6789/0 conn(0x7f50e80a8970 legacy=0x7f50e813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:14.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 10 monclient: _finish_hunting 0 2023-12-07T23:46:14.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 1 monclient: found mon.noname-a 2023-12-07T23:46:14.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:14.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:14.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.340+0000 7f50e759e640 1 -- 172.21.15.129:0/4028031532 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e80aaea0 con 0x7f50e813f580 2023-12-07T23:46:14.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.366+0000 7f50e759e640 1 --2- 172.21.15.129:0/4028031532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e813f580 0x7f50e81426e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009e50 tx=0x7f50e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36833d79ea65ee7b server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:14.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 1 -- 172.21.15.129:0/4028031532 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e002c9e0 con 0x7f50e813f580 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: dump: 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:14.464 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: _finish_auth 0 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: _check_auth_tickets 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 1 -- 172.21.15.129:0/4028031532 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e002cb40 con 0x7f50e813f580 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 1 -- 172.21.15.129:0/4028031532 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e00343f0 con 0x7f50e813f580 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50ed0c9640 5 monclient: authenticate success, global_id 4257 2023-12-07T23:46:14.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50ed0c9640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50ed0c9640 4 set_mon_vals no callback set 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.460+0000 7f50e659c640 10 monclient: dump: 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:14.466 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:14.467 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:14.467 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 10 monclient: shutdown 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 -- 172.21.15.129:0/4028031532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e813f580 msgr2=0x7f50e81426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/4028031532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e813f580 0x7f50e81426e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009e50 tx=0x7f50e002be40 comp rx=0 tx=0).stop 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 -- 172.21.15.129:0/4028031532 shutdown_connections 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/4028031532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e813f580 0x7f50e81426e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/4028031532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e81421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 -- 172.21.15.129:0/4028031532 >> 172.21.15.129:0/4028031532 conn(0x7f50e8066dd0 msgr2=0x7f50e80b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 -- 172.21.15.129:0/4028031532 shutdown_connections 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.461+0000 7f50ed0c9640 1 -- 172.21.15.129:0/4028031532 wait complete. 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 10 monclient: build_initial_monmap 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 10 monclient: monmap: 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:14.472 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 1 librados: starting msgr at 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 1 librados: starting objecter 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 1 Processor -- start 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 1 -- start start 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 1 librados: setting wanted keys 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 1 librados: calling monclient init 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.462+0000 7f50ed0c9640 10 monclient: init 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e80a8970 0x7f50e813abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): picked mon.noname-a con 0x7f50e80a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e813b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): picked mon.noname-c con 0x7f50e80a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50e813f580 0x7f50e81daa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): picked mon.noname-b con 0x7f50e813f580 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e81da520 con 0x7f50e80a8970 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8137d70 con 0x7f50e813f580 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8137f30 con 0x7f50e80a9bd0 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.463+0000 7f50ed0c9640 10 monclient(hunting): authenticate will time out at 1374.651270s 2023-12-07T23:46:14.475 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:10.941 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:14.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.483+0000 7f50e6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e80a8970 0x7f50e813abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:14.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.484+0000 7f50e6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e80a8970 0x7f50e813abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42728/0 (socket says 172.21.15.129:42728) 2023-12-07T23:46:14.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.484+0000 7f50e6d9d640 1 -- 172.21.15.129:0/2608190747 learned_addr learned my addr 172.21.15.129:0/2608190747 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:14.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.484+0000 7f50e6d9d640 10 monclient(hunting): get_auth_request con 0x7f50e80a8970 auth_method 0 2023-12-07T23:46:14.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.484+0000 7f50e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:14.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.484+0000 7f50e6d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:14.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.484+0000 7f50e6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.511+0000 7f50e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.511+0000 7f50e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.511+0000 7f50e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:14.521 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.519+0000 7f50dffff640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e813b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:14.524 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.522+0000 7f50dffff640 10 monclient(hunting): get_auth_request con 0x7f50e80a9bd0 auth_method 0 2023-12-07T23:46:14.525 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.522+0000 7f50dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:14.525 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.522+0000 7f50dffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:14.525 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.522+0000 7f50dffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:14.540 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.538+0000 7f50dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:14.541 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.538+0000 7f50dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:14.541 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.538+0000 7f50dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 1 -- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50e813f580 msgr2=0x7f50e81daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50e813f580 0x7f50e81daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 1 -- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e80a8970 msgr2=0x7f50e813abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e80a8970 0x7f50e813abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 1 monclient: found mon.noname-c 2023-12-07T23:46:14.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:14.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:14.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.541+0000 7f50dffff640 1 -- 172.21.15.129:0/2608190747 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e0009b30 con 0x7f50e80a9bd0 2023-12-07T23:46:14.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.619+0000 7f50dffff640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e813b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f50d000b3c0 tx=0x7f50d000b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:14.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.619+0000 7f50df7fe640 10 client.?.objecter ms_handle_connect 0x7f50e80a9bd0 2023-12-07T23:46:14.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.619+0000 7f50df7fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:14.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 1 -- 172.21.15.129:0/2608190747 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0011830 con 0x7f50e80a9bd0 2023-12-07T23:46:14.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: dump: 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:14.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 5 monclient: authenticate success, global_id 4235 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 1 -- 172.21.15.129:0/2608190747 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d0011990 con 0x7f50e80a9bd0 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 1 -- 172.21.15.129:0/2608190747 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0011d40 con 0x7f50e80a9bd0 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50df7fe640 10 monclient: dump: 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:14.676 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50e5d9b640 4 set_mon_vals no callback set 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 10 monclient: _renew_subs 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e813a690 con 0x7f50e80a9bd0 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 10 monclient: _renew_subs 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.668+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e81daf40 con 0x7f50e80a9bd0 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.669+0000 7f50ed0c9640 1 librados: init done 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.669+0000 7f50ed0c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.669+0000 7f50ed0c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.669+0000 7f50ed0c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.669+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e81da380 con 0x7f50e80a9bd0 2023-12-07T23:46:14.921 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.918+0000 7f50df7fe640 1 -- 172.21.15.129:0/2608190747 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f50d001a400 con 0x7f50e80a9bd0 2023-12-07T23:46:14.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.918+0000 7f50df7fe640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f50c806c870 0x7f50c806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:14.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.919+0000 7f50e6d9d640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f50c806c870 0x7f50c806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:14.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.919+0000 7f50e6d9d640 10 monclient: get_auth_request con 0x7f50c806c870 auth_method 0 2023-12-07T23:46:14.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.919+0000 7f50e6d9d640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f50c806c870 0x7f50c806ed30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50d8001840 tx=0x7f50d8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:14.923 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.919+0000 7f50df7fe640 10 client.4235.objecter ms_handle_connect 0x7f50c806c870 2023-12-07T23:46:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 1 -- 172.21.15.129:0/2608190747 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f50d0058ec0 con 0x7f50e80a9bd0 2023-12-07T23:46:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 10 client.4235.objecter ms_dispatch 0x7f50e8000b90 osd_map(19..19 src has 1..19) v4 2023-12-07T23:46:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 3 client.4235.objecter handle_osd_map got epochs [19,19] > 0 2023-12-07T23:46:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 3 client.4235.objecter handle_osd_map decoding full epoch 19 2023-12-07T23:46:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 20 client.4235.objecter dump_active .. 0 homeless 2023-12-07T23:46:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 1 -- 172.21.15.129:0/2608190747 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50d0054b00 con 0x7f50e80a9bd0 2023-12-07T23:46:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:14.953+0000 7f50df7fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:15.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.150+0000 7f50ed0c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:15.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.150+0000 7f50ed0c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:15.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.150+0000 7f50ed0c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:15.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.150+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50e81da380 con 0x7f50e80a9bd0 2023-12-07T23:46:15.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.300+0000 7f50df7fe640 1 -- 172.21.15.129:0/2608190747 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14663 (secure 0 0 0) 0x7f50e81da380 con 0x7f50e80a9bd0 2023-12-07T23:46:15.304 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.300+0000 7f50df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:15.304 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.300+0000 7f50df7fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:15.304 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:46:15.304 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:46:14.085997+0000","last_up_change":"2023-12-07T23:46:04.801550+0000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T23:46:02.095929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T23:46:10.536846+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6825","nonce":771847878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6827","nonce":771847878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6831","nonce":771847878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6829","nonce":771847878}]},"public_addr":"172.21.15.129:6825/771847878","cluster_addr":"172.21.15.129:6827/771847878","heartbeat_back_addr":"172.21.15.129:6831/771847878","heartbeat_front_addr":"172.21.15.129:6829/771847878","state":["exists","up"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6817","nonce":3964479910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6819","nonce":3964479910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6823","nonce":3964479910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6821","nonce":3964479910}]},"public_addr":"172.21.15.129:6817/3964479910","cluster_addr":"172.21.15.129:6819/3964479910","heartbeat_back_addr":"172.21.15.129:6823/3964479910","heartbeat_front_addr":"172.21.15.129:6821/3964479910","state":["exists","up"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6801","nonce":262618579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6803","nonce":262618579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6807","nonce":262618579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6805","nonce":262618579}]},"public_addr":"172.21.15.129:6801/262618579","cluster_addr":"172.21.15.129:6803/262618579","heartbeat_back_addr":"172.21.15.129:6807/262618579","heartbeat_front_addr":"172.21.15.129:6805/262618579","state":["exists","up"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6809","nonce":2348435352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6811","nonce":2348435352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6815","nonce":2348435352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6813","nonce":2348435352}]},"public_addr":"172.21.15.129:6809/2348435352","cluster_addr":"172.21.15.129:6811/2348435352","heartbeat_back_addr":"172.21.15.129:6815/2348435352","heartbeat_front_addr":"172.21.15.129:6813/2348435352","state":["exists","up"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6827","nonce":917696104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6829","nonce":917696104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6833","nonce":917696104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6831","nonce":917696104}]},"public_addr":"172.21.15.184:6827/917696104","cluster_addr":"172.21.15.184:6829/917696104","heartbeat_back_addr":"172.21.15.184:6833/917696104","heartbeat_front_addr":"172.21.15.184:6831/917696104","state":["exists","up"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6819","nonce":4213359653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6821","nonce":4213359653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6825","nonce":4213359653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6823","nonce":4213359653}]},"public_addr":"172.21.15.184:6819/4213359653","cluster_addr":"172.21.15.184:6821/4213359653","heartbeat_back_addr":"172.21.15.184:6825/4213359653","heartbeat_front_addr":"172.21.15.184:6823/4213359653","state":["exists","up"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6807","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6809","nonce":105483051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6813","nonce":105483051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6817","nonce":105483051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6815","nonce":105483051}]},"public_addr":"172.21.15.184:6809/105483051","cluster_addr":"172.21.15.184:6813/105483051","heartbeat_back_addr":"172.21.15.184:6817/105483051","heartbeat_front_addr":"172.21.15.184:6815/105483051","state":["exists","up"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6803","nonce":59760990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6805","nonce":59760990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6811","nonce":59760990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6808","nonce":59760990}]},"public_addr":"172.21.15.184:6803/59760990","cluster_addr":"172.21.15.184:6805/59760990","heartbeat_back_addr":"172.21.15.184:6811/59760990","heartbeat_front_addr":"172.21.15.184:6808/59760990","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.382162+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:00.490549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.542980+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.347947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.274789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.915445+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.493984+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.330992+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:46:15.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 10 librados: watch_flush enter 2023-12-07T23:46:15.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 10 librados: watch_flush exit 2023-12-07T23:46:15.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 20 client.4235.objecter shutdown clearing up homeless session... 2023-12-07T23:46:15.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 10 client.4235.objecter successfully canceled tick 2023-12-07T23:46:15.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f50c806c870 msgr2=0x7f50c806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f50c806c870 0x7f50c806ed30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50d8001840 tx=0x7f50d8009040 comp rx=0 tx=0).stop 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 10 monclient: shutdown 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 msgr2=0x7f50e813b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e813b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f50d000b3c0 tx=0x7f50d000b880 comp rx=0 tx=0).stop 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 shutdown_connections 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f50c806c870 0x7f50c806ed30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50e813f580 0x7f50e81daa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50e80a9bd0 0x7f50e813b380 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 --2- 172.21.15.129:0/2608190747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e80a8970 0x7f50e813abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.302+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 >> 172.21.15.129:0/2608190747 conn(0x7f50e8066dd0 msgr2=0x7f50e813ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.303+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 shutdown_connections 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.303+0000 7f50ed0c9640 1 -- 172.21.15.129:0/2608190747 wait complete. 2023-12-07T23:46:15.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.303+0000 7f50ed0c9640 1 librados: shutdown 2023-12-07T23:46:15.317 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-07T23:46:15.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.456+0000 7f1d827dc640 10 monclient: get_monmap_and_config 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.456+0000 7f1d827dc640 10 monclient: build_initial_monmap 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.456+0000 7f1d827dc640 10 monclient: monmap: 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:15.457273+0000 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:15.457273+0000 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:15.460 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:15.460 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:15.460 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:15.460 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:15.460 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.456+0000 7f1d827dc640 1 Processor -- start 2023-12-07T23:46:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.456+0000 7f1d827dc640 1 -- start start 2023-12-07T23:46:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.456+0000 7f1d827dc640 10 monclient: init 2023-12-07T23:46:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-12-07T23:46:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d7c0a8970 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c0a9bd0 0x7f1d7c1421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7c0a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d7c13f580 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c0aa950 con 0x7f1d7c0a8970 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c0a90e0 con 0x7f1d7c13f580 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c142df0 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.457+0000 7f1d827dc640 10 monclient(hunting): authenticate will time out at 1375.645592s 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.458+0000 7f1d7b7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c0a9bd0 0x7f1d7c1421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:15.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.458+0000 7f1d7bfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f1d7c0a8970 0x7f1d7c13ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43092/0 (socket says 172.21.15.129:43092) 2023-12-07T23:46:15.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.458+0000 7f1d7bfff640 1 -- 172.21.15.129:0/140802569 learned_addr learned my addr 172.21.15.129:0/140802569 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:15.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.458+0000 7f1d7b7fe640 10 monclient(hunting): get_auth_request con 0x7f1d7c0a9bd0 auth_method 0 2023-12-07T23:46:15.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.458+0000 7f1d7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:15.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.458+0000 7f1d7b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:15.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.458+0000 7f1d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:15.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.459+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:15.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.459+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:15.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.459+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:15.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.460+0000 7f1d7affd640 1 -- 172.21.15.129:0/140802569 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768175191 0 0) 0x7f1d7c0a90e0 con 0x7f1d7c13f580 2023-12-07T23:46:15.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.460+0000 7f1d7affd640 10 monclient(hunting): _negotiate 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.460+0000 7f1d7affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.460+0000 7f1d7affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.460+0000 7f1d7affd640 10 monclient(hunting): my global_id is 4285 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.460+0000 7f1d7affd640 1 -- 172.21.15.129:0/140802569 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d64003520 con 0x7f1d7c13f580 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 1 -- 172.21.15.129:0/140802569 >> v1:172.21.15.184:6789/0 conn(0x7f1d7c13f580 legacy=0x7f1d7c1426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 1 -- 172.21.15.129:0/140802569 >> v1:172.21.15.129:6789/0 conn(0x7f1d7c0a8970 legacy=0x7f1d7c13ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 1 monclient: found mon.noname-b 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:15.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:15.467 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.461+0000 7f1d7b7fe640 1 -- 172.21.15.129:0/140802569 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0aabc0 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.467 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.463+0000 7f1d7b7fe640 1 --2- 172.21.15.129:0/140802569 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c0a9bd0 0x7f1d7c1421a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1d700097d0 tx=0x7f1d7002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58c19ebe3c34c035 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 1 -- 172.21.15.129:0/140802569 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70031770 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: dump: 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: _finish_auth 0 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 1 -- 172.21.15.129:0/140802569 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d700318d0 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 1 -- 172.21.15.129:0/140802569 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70031c60 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d7affd640 10 monclient: dump: 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d827dc640 5 monclient: authenticate success, global_id 4238 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d827dc640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.479+0000 7f1d827dc640 4 set_mon_vals no callback set 2023-12-07T23:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 10 monclient: shutdown 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 1 -- 172.21.15.129:0/140802569 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c0a9bd0 msgr2=0x7f1d7c1421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 1 --2- 172.21.15.129:0/140802569 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c0a9bd0 0x7f1d7c1421a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1d700097d0 tx=0x7f1d7002b810 comp rx=0 tx=0).stop 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 1 -- 172.21.15.129:0/140802569 shutdown_connections 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 1 --2- 172.21.15.129:0/140802569 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c0a9bd0 0x7f1d7c1421a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 1 -- 172.21.15.129:0/140802569 >> 172.21.15.129:0/140802569 conn(0x7f1d7c066dd0 msgr2=0x7f1d7c0b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 1 -- 172.21.15.129:0/140802569 shutdown_connections 2023-12-07T23:46:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 1 -- 172.21.15.129:0/140802569 wait complete. 2023-12-07T23:46:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 10 monclient: build_initial_monmap 2023-12-07T23:46:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.480+0000 7f1d827dc640 10 monclient: monmap: 2023-12-07T23:46:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.481+0000 7f1d827dc640 1 librados: starting msgr at 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.481+0000 7f1d827dc640 1 librados: starting objecter 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.481+0000 7f1d827dc640 1 Processor -- start 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.481+0000 7f1d827dc640 1 -- start start 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.481+0000 7f1d827dc640 1 librados: setting wanted keys 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.481+0000 7f1d827dc640 1 librados: calling monclient init 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.481+0000 7f1d827dc640 10 monclient: init 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:46:15.489 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d7c0a8970 0x7f1d7c0ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): picked mon.noname-a con 0x7f1d7c0a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d7c0a9bd0 0x7f1d7c0ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7c0a9bd0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c13f580 0x7f1d7c135890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): picked mon.noname-c con 0x7f1d7c13f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0acb10 con 0x7f1d7c0a8970 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0ada30 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c139240 con 0x7f1d7c13f580 2023-12-07T23:46:15.490 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d827dc640 10 monclient(hunting): authenticate will time out at 1375.670058s 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d7b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d7c0a9bd0 0x7f1d7c0ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.482+0000 7f1d7bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d7c0a8970 0x7f1d7c0ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d7c0a8970 0x7f1d7c0ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42742/0 (socket says 172.21.15.129:42742) 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 1 -- 172.21.15.129:0/436958921 learned_addr learned my addr 172.21.15.129:0/436958921 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 10 monclient(hunting): get_auth_request con 0x7f1d7c0a8970 auth_method 0 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7b7fe640 10 monclient(hunting): get_auth_request con 0x7f1d7c0a9bd0 auth_method 0 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:15.491 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:15.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:15.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:15.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:15.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.483+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:15.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 1 -- 172.21.15.129:0/436958921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c13f580 msgr2=0x7f1d7c135890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c13f580 0x7f1d7c135890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 1 -- 172.21.15.129:0/436958921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d7c0a8970 msgr2=0x7f1d7c0ac090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d7c0a8970 0x7f1d7c0ac090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 1 monclient: found mon.noname-b 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.486+0000 7f1d7b7fe640 1 -- 172.21.15.129:0/436958921 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d70009400 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.488+0000 7f1d7b7fe640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d7c0a9bd0 0x7f1d7c0ac5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0b51f0 tx=0x7f1d700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.488+0000 7f1d78ff9640 10 client.?.objecter ms_handle_connect 0x7f1d7c0a9bd0 2023-12-07T23:46:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.488+0000 7f1d78ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:15.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 1 -- 172.21.15.129:0/436958921 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70032810 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: dump: 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: _finish_auth 0 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 1 -- 172.21.15.129:0/436958921 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d70032990 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 5 monclient: authenticate success, global_id 4288 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 10 monclient: _renew_subs 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c0ab620 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 1 -- 172.21.15.129:0/436958921 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70032d40 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d78ff9640 10 monclient: dump: 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d7a7fc640 4 set_mon_vals no callback set 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 10 monclient: _renew_subs 2023-12-07T23:46:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c135fb0 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.492+0000 7f1d827dc640 1 librados: init done 2023-12-07T23:46:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.493+0000 7f1d827dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.493+0000 7f1d827dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.493+0000 7f1d827dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.493+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c04e840 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.731 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.728+0000 7f1d78ff9640 1 -- 172.21.15.129:0/436958921 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1d7003b3f0 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.729+0000 7f1d78ff9640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1d6806c870 0x7f1d6806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:15.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.729+0000 7f1d7bfff640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1d6806c870 0x7f1d6806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:15.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.729+0000 7f1d7bfff640 10 monclient: get_auth_request con 0x7f1d6806c870 auth_method 0 2023-12-07T23:46:15.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.729+0000 7f1d78ff9640 10 client.4288.objecter ms_handle_connect 0x7f1d6806c870 2023-12-07T23:46:15.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.729+0000 7f1d7bfff640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1d6806c870 0x7f1d6806ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c004e40 tx=0x7f1d6c007f10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.732+0000 7f1d78ff9640 1 -- 172.21.15.129:0/436958921 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1d7007ae70 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.732+0000 7f1d78ff9640 10 client.4288.objecter ms_dispatch 0x7f1d7c000b90 osd_map(20..20 src has 1..20) v4 2023-12-07T23:46:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.732+0000 7f1d78ff9640 3 client.4288.objecter handle_osd_map got epochs [20,20] > 0 2023-12-07T23:46:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.732+0000 7f1d78ff9640 3 client.4288.objecter handle_osd_map decoding full epoch 20 2023-12-07T23:46:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.732+0000 7f1d78ff9640 20 client.4288.objecter dump_active .. 0 homeless 2023-12-07T23:46:15.744 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.741+0000 7f1d78ff9640 1 -- 172.21.15.129:0/436958921 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d70076ab0 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.744 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.742+0000 7f1d78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:15.744 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.742+0000 7f1d78ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:15.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.923+0000 7f1d827dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:15.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.923+0000 7f1d827dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:15.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.923+0000 7f1d827dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:15.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.923+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d7c139380 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.955+0000 7f1d78ff9640 1 -- 172.21.15.129:0/436958921 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14650 (secure 0 0 0) 0x7f1d7007a0d0 con 0x7f1d7c0a9bd0 2023-12-07T23:46:15.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.955+0000 7f1d78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T23:46:15.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.955+0000 7f1d78ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:15.958 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:46:15.959 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-07T23:46:15.655954+0000","last_up_change":"2023-12-07T23:46:04.801550+0000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T23:46:02.095929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T23:46:10.536846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6825","nonce":771847878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6827","nonce":771847878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6831","nonce":771847878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6829","nonce":771847878}]},"public_addr":"172.21.15.129:6825/771847878","cluster_addr":"172.21.15.129:6827/771847878","heartbeat_back_addr":"172.21.15.129:6831/771847878","heartbeat_front_addr":"172.21.15.129:6829/771847878","state":["exists","up"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6817","nonce":3964479910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6819","nonce":3964479910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6823","nonce":3964479910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6821","nonce":3964479910}]},"public_addr":"172.21.15.129:6817/3964479910","cluster_addr":"172.21.15.129:6819/3964479910","heartbeat_back_addr":"172.21.15.129:6823/3964479910","heartbeat_front_addr":"172.21.15.129:6821/3964479910","state":["exists","up"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6801","nonce":262618579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6803","nonce":262618579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6807","nonce":262618579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6805","nonce":262618579}]},"public_addr":"172.21.15.129:6801/262618579","cluster_addr":"172.21.15.129:6803/262618579","heartbeat_back_addr":"172.21.15.129:6807/262618579","heartbeat_front_addr":"172.21.15.129:6805/262618579","state":["exists","up"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6809","nonce":2348435352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6811","nonce":2348435352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6815","nonce":2348435352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6813","nonce":2348435352}]},"public_addr":"172.21.15.129:6809/2348435352","cluster_addr":"172.21.15.129:6811/2348435352","heartbeat_back_addr":"172.21.15.129:6815/2348435352","heartbeat_front_addr":"172.21.15.129:6813/2348435352","state":["exists","up"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6827","nonce":917696104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6829","nonce":917696104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6833","nonce":917696104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6831","nonce":917696104}]},"public_addr":"172.21.15.184:6827/917696104","cluster_addr":"172.21.15.184:6829/917696104","heartbeat_back_addr":"172.21.15.184:6833/917696104","heartbeat_front_addr":"172.21.15.184:6831/917696104","state":["exists","up"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6819","nonce":4213359653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6821","nonce":4213359653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6825","nonce":4213359653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6823","nonce":4213359653}]},"public_addr":"172.21.15.184:6819/4213359653","cluster_addr":"172.21.15.184:6821/4213359653","heartbeat_back_addr":"172.21.15.184:6825/4213359653","heartbeat_front_addr":"172.21.15.184:6823/4213359653","state":["exists","up"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6807","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6809","nonce":105483051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6813","nonce":105483051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6817","nonce":105483051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6815","nonce":105483051}]},"public_addr":"172.21.15.184:6809/105483051","cluster_addr":"172.21.15.184:6813/105483051","heartbeat_back_addr":"172.21.15.184:6817/105483051","heartbeat_front_addr":"172.21.15.184:6815/105483051","state":["exists","up"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6803","nonce":59760990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6805","nonce":59760990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6811","nonce":59760990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6808","nonce":59760990}]},"public_addr":"172.21.15.184:6803/59760990","cluster_addr":"172.21.15.184:6805/59760990","heartbeat_back_addr":"172.21.15.184:6811/59760990","heartbeat_front_addr":"172.21.15.184:6808/59760990","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.382162+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:00.490549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.542980+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.347947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.274789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.915445+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.493984+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.330992+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T23:46:15.960 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 10 librados: watch_flush enter 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 10 librados: watch_flush exit 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 20 client.4288.objecter shutdown clearing up homeless session... 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 10 client.4288.objecter successfully canceled tick 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1d6806c870 msgr2=0x7f1d6806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1d6806c870 0x7f1d6806ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c004e40 tx=0x7f1d6c007f10 comp rx=0 tx=0).stop 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 10 monclient: shutdown 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d7c0a9bd0 msgr2=0x7f1d7c0ac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d7c0a9bd0 0x7f1d7c0ac5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0b51f0 tx=0x7f1d700046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 shutdown_connections 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1d6806c870 0x7f1d6806ed30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1d7c13f580 0x7f1d7c135890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d7c0a9bd0 0x7f1d7c0ac5d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 --2- 172.21.15.129:0/436958921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d7c0a8970 0x7f1d7c0ac090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:15.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 >> 172.21.15.129:0/436958921 conn(0x7f1d7c066dd0 msgr2=0x7f1d7c0b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:15.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 shutdown_connections 2023-12-07T23:46:15.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 -- 172.21.15.129:0/436958921 wait complete. 2023-12-07T23:46:15.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:15.957+0000 7f1d827dc640 1 librados: shutdown 2023-12-07T23:46:15.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-07T23:46:15.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-07T23:46:15.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-07T23:46:15.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-07T23:46:15.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-07T23:46:15.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-07T23:46:15.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-07T23:46:15.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-07T23:46:16.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.149+0000 7ff7b16b4640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.149+0000 7ff7b16b4640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.149+0000 7ff7b16b4640 10 monclient: monmap: 2023-12-07T23:46:16.152 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.152 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.150388+0000 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.150388+0000 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.153 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7fcf5bfff640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7fcf5bfff640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7fcf5bfff640 10 monclient: monmap: 2023-12-07T23:46:16.160 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.158262+0000 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.158262+0000 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.162 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.150+0000 7ff7b16b4640 1 Processor -- start 2023-12-07T23:46:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.150+0000 7ff7b16b4640 1 -- start start 2023-12-07T23:46:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.150+0000 7ff7b16b4640 10 monclient: init 2023-12-07T23:46:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a40391f0 0x7ff7a40c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient(hunting): picked mon.noname-b con 0x7ff7a40391f0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7a4034630 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff7a4033d30 0x7ff7a40d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient(hunting): picked mon.noname-c con 0x7ff7a4033d30 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a40c80e0 con 0x7ff7a4034630 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a40c55c0 con 0x7ff7a4033d30 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a4037620 con 0x7ff7a40391f0 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.151+0000 7ff7b16b4640 10 monclient(hunting): authenticate will time out at 1376.339175s 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.152+0000 7ff7aaffd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a40391f0 0x7ff7a40c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.154+0000 7ff7aa7fc640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff7a4034630 0x7ff7a4033620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43108/0 (socket says 172.21.15.129:43108) 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.154+0000 7ff7aa7fc640 1 -- 172.21.15.129:0/3201794464 learned_addr learned my addr 172.21.15.129:0/3201794464 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.154+0000 7ff7aaffd640 10 monclient(hunting): get_auth_request con 0x7ff7a40391f0 auth_method 0 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.154+0000 7ff7aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.154+0000 7ff7aaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.154+0000 7ff7aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.155+0000 7ff7abfff640 1 --2- 172.21.15.129:0/3201794464 >> v2:172.21.15.184:3300/0 conn(0x7ff7a4033d30 0x7ff7a40d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7ff7abfff640 10 monclient(hunting): get_auth_request con 0x7ff7a4033d30 auth_method 0 2023-12-07T23:46:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7ff7abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7ff7abfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7ff7abfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.157+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.158+0000 7fcf5bfff640 1 Processor -- start 2023-12-07T23:46:16.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.158+0000 7fcf5bfff640 1 -- start start 2023-12-07T23:46:16.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.158+0000 7fcf5bfff640 10 monclient: init 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.158+0000 7fcf5bfff640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fcf54006c10 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcf5402ea60 0x7fcf5402e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fcf5402ea60 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcf5402e630 0x7fcf54032880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fcf5402e630 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf54006ff0 con 0x7fcf5402e630 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf5402fab0 con 0x7fcf5402ea60 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5402c7f0 con 0x7fcf54006c10 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.159+0000 7fcf5bfff640 10 monclient(hunting): authenticate will time out at 1376.347083s 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.160+0000 7fcf5a7fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcf5402ea60 0x7fcf5402e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.160+0000 7fe4225fb640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.160+0000 7fe4225fb640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.160+0000 7fe4225fb640 10 monclient: monmap: 2023-12-07T23:46:16.172 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.161683+0000 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.161683+0000 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.173 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcf5402e630 0x7fcf54032880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcf5402e630 0x7fcf54032880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42756/0 (socket says 172.21.15.129:42756) 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5b7fe640 1 -- 172.21.15.129:0/2070174293 learned_addr learned my addr 172.21.15.129:0/2070174293 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5b7fe640 10 monclient(hunting): get_auth_request con 0x7fcf5402e630 auth_method 0 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5a7fc640 10 monclient(hunting): get_auth_request con 0x7fcf5402ea60 auth_method 0 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fe4225fb640 1 Processor -- start 2023-12-07T23:46:16.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fe4225fb640 1 -- start start 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fe4225fb640 10 monclient: init 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5affd640 1 --2- 172.21.15.129:0/2070174293 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5affd640 10 monclient(hunting): get_auth_request con 0x7fcf54006c10 auth_method 0 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7fcf5affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7ff7abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7ff7abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.161+0000 7ff7abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 1 -- 172.21.15.129:0/3201794464 >> v2:172.21.15.184:3300/0 conn(0x7ff7a4033d30 msgr2=0x7ff7a40d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 1 --2- 172.21.15.129:0/3201794464 >> v2:172.21.15.184:3300/0 conn(0x7ff7a4033d30 0x7ff7a40d2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 1 -- 172.21.15.129:0/3201794464 >> v1:172.21.15.129:6789/0 conn(0x7ff7a4034630 legacy=0x7ff7a4033620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7ff7aaffd640 1 -- 172.21.15.129:0/3201794464 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40c8ae0 con 0x7ff7a40391f0 2023-12-07T23:46:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fcf5a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fcf5a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fcf5a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe414006fa0 0x7fe4140cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe414006fa0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe414007380 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe414033900 0x7fe4140d24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe414033900 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4140357b0 con 0x7fe414007380 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4140c5860 con 0x7fe414033900 2023-12-07T23:46:16.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe414036630 con 0x7fe414006fa0 2023-12-07T23:46:16.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7fe4225fb640 10 monclient(hunting): authenticate will time out at 1376.350818s 2023-12-07T23:46:16.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7efd0968a640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.162+0000 7efd0968a640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.163+0000 7efd0968a640 10 monclient: monmap: 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.164169+0000 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.164169+0000 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.163+0000 7fcf5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.163+0000 7fcf5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.163+0000 7fcf5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.163+0000 7fcf5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.163+0000 7fcf5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.163+0000 7fcf5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.164+0000 7fe7515a9640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.164+0000 7fe7515a9640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.164+0000 7fe7515a9640 10 monclient: monmap: 2023-12-07T23:46:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.165244+0000 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.165244+0000 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.186 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.164+0000 7efd0968a640 1 Processor -- start 2023-12-07T23:46:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.164+0000 7efd0968a640 1 -- start start 2023-12-07T23:46:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.164+0000 7efd0968a640 10 monclient: init 2023-12-07T23:46:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fe7515a9640 1 Processor -- start 2023-12-07T23:46:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fe7515a9640 1 -- start start 2023-12-07T23:46:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fe7515a9640 10 monclient: init 2023-12-07T23:46:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-12-07T23:46:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd040aee70 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efd0413f470 0x7efd0406e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 1 -- 172.21.15.129:0/2070174293 >> v2:172.21.15.184:3300/0 conn(0x7fcf5402ea60 msgr2=0x7fcf5402e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 1 --2- 172.21.15.129:0/2070174293 >> v2:172.21.15.184:3300/0 conn(0x7fcf5402ea60 0x7fcf5402e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 1 -- 172.21.15.129:0/2070174293 >> v2:172.21.15.129:3300/0 conn(0x7fcf5402e630 msgr2=0x7fcf54032880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 1 --2- 172.21.15.129:0/2070174293 >> v2:172.21.15.129:3300/0 conn(0x7fcf5402e630 0x7fcf54032880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fcf5affd640 1 -- 172.21.15.129:0/2070174293 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5402c480 con 0x7fcf54006c10 2023-12-07T23:46:16.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 10 monclient(hunting): picked mon.noname-c con 0x7efd0413f470 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fe7515a9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7fe7515a9640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-12-07T23:46:16.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe744006400 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe744032020 0x7fe7440c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd04145630 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0406fac0 con 0x7efd04145630 2023-12-07T23:46:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.165+0000 7efd0968a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0406fda0 con 0x7efd040aee70 2023-12-07T23:46:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7efd0968a640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7efd0968a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd040af250 con 0x7efd0413f470 2023-12-07T23:46:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7efd0968a640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7efd0968a640 10 monclient(hunting): authenticate will time out at 1376.354035s 2023-12-07T23:46:16.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 10 monclient(hunting): picked mon.noname-c con 0x7fe744032020 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe7440c53e0 0x7fe7440d24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 10 monclient(hunting): picked mon.noname-a con 0x7fe7440c53e0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe744033950 con 0x7fe744006400 2023-12-07T23:46:16.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7440067e0 con 0x7fe7440c53e0 2023-12-07T23:46:16.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.192 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe744033250 con 0x7fe744032020 2023-12-07T23:46:16.192 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.192 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7fe7515a9640 10 monclient(hunting): authenticate will time out at 1376.354327s 2023-12-07T23:46:16.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7f2b9be3f640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7f2b9be3f640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.166+0000 7f2b9be3f640 10 monclient: monmap: 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.167632+0000 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.167632+0000 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.194 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.167+0000 7f2b9be3f640 1 Processor -- start 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.167+0000 7f2b9be3f640 1 -- start start 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.167+0000 7f2b9be3f640 10 monclient: init 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b940a8970 0x7f2b9413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): picked mon.noname-b con 0x7f2b940a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2b940a9bd0 0x7f2b9413f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b940a9bd0 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2b941421a0 0x7f2b94142630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b941421a0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b940aa950 con 0x7f2b941421a0 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b940a90e0 con 0x7f2b940a9bd0 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b940aabc0 con 0x7f2b940a8970 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7f2b9be3f640 10 monclient(hunting): authenticate will time out at 1376.356530s 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7efd037fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efd0413f470 0x7efd0406e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7efd037fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efd0413f470 0x7efd0406e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40452/0 (socket says 172.21.15.129:40452) 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7efd037fe640 1 -- 172.21.15.129:0/201858483 learned_addr learned my addr 172.21.15.129:0/201858483 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7efd037fe640 10 monclient(hunting): get_auth_request con 0x7efd0413f470 auth_method 0 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7efd037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7efd037fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.168+0000 7efd037fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.169+0000 7fcf5affd640 1 --2- 172.21.15.129:0/2070174293 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540cf5e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcf540076f0 tx=0x7fcf5002b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f22f87f4545334bc server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.169+0000 7ff7aaffd640 1 --2- 172.21.15.129:0/3201794464 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a40391f0 0x7ff7a40c85a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0097a0 tx=0x7ff79c02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63cc890783402148 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.169+0000 7fe421dfa640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe414033900 0x7fe4140d24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.169+0000 7efd037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.169+0000 7efd037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.169+0000 7efd037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.170+0000 7fe421dfa640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe414033900 0x7fe4140d24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42770/0 (socket says 172.21.15.129:42770) 2023-12-07T23:46:16.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.170+0000 7fe421dfa640 1 -- 172.21.15.129:0/4254535348 learned_addr learned my addr 172.21.15.129:0/4254535348 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.170+0000 7fe421dfa640 10 monclient(hunting): get_auth_request con 0x7fe414033900 auth_method 0 2023-12-07T23:46:16.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.170+0000 7fe421dfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.170+0000 7fe421dfa640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.170+0000 7fe421dfa640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.170+0000 7fe74b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe7440c53e0 0x7fe7440d24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2023-12-07T23:46:16.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 1 -- 172.21.15.129:0/201858483 >> v1:172.21.15.184:6789/0 conn(0x7efd040aee70 legacy=0x7efd0413ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 1 -- 172.21.15.129:0/201858483 >> v1:172.21.15.129:6789/0 conn(0x7efd04145630 legacy=0x7efd04145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7efd037fe640 1 -- 172.21.15.129:0/201858483 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd04070080 con 0x7efd0413f470 2023-12-07T23:46:16.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7fe421dfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7fe421dfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7fe421dfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.171+0000 7fe74a7fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe744032020 0x7fe7440c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2023-12-07T23:46:16.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 1 -- 172.21.15.129:0/4254535348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe414006fa0 msgr2=0x7fe4140cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 1 --2- 172.21.15.129:0/4254535348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe414006fa0 0x7fe4140cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 1 -- 172.21.15.129:0/4254535348 >> v1:172.21.15.129:6789/0 conn(0x7fe414007380 legacy=0x7fe414035f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.172+0000 7fe421dfa640 1 -- 172.21.15.129:0/4254535348 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe414035a90 con 0x7fe414033900 2023-12-07T23:46:16.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74a7fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe744032020 0x7fe7440c4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40464/0 (socket says 172.21.15.129:40464) 2023-12-07T23:46:16.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74a7fc640 1 -- 172.21.15.129:0/1880544313 learned_addr learned my addr 172.21.15.129:0/1880544313 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74a7fc640 10 monclient(hunting): get_auth_request con 0x7fe744032020 auth_method 0 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74b7fe640 10 monclient(hunting): get_auth_request con 0x7fe7440c53e0 auth_method 0 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.173+0000 7fe74b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.174+0000 7fe74a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.174+0000 7fe74a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.174+0000 7fe74a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.174+0000 7f2b9a3dc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2b941421a0 0x7f2b94142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2023-12-07T23:46:16.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 1 -- 172.21.15.129:0/1880544313 >> v2:172.21.15.129:3300/0 conn(0x7fe7440c53e0 msgr2=0x7fe7440d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 1 --2- 172.21.15.129:0/1880544313 >> v2:172.21.15.129:3300/0 conn(0x7fe7440c53e0 0x7fe7440d24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 1 -- 172.21.15.129:0/1880544313 >> v1:172.21.15.184:6789/0 conn(0x7fe744006400 legacy=0x7fe744031910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.175+0000 7fe74a7fc640 1 -- 172.21.15.129:0/1880544313 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe744033530 con 0x7fe744032020 2023-12-07T23:46:16.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.176+0000 7f2b993da640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2b940a9bd0 0x7f2b9413f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.176+0000 7fe421dfa640 1 --2- 172.21.15.129:0/4254535348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 0x7fe4140d24b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe418009870 tx=0x7fe41802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87359481807d882b server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7f2b9a3dc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2b941421a0 0x7f2b94142630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42784/0 (socket says 172.21.15.129:42784) 2023-12-07T23:46:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7f2b9a3dc640 1 -- 172.21.15.129:0/2333347620 learned_addr learned my addr 172.21.15.129:0/2333347620 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7f2b9a3dc640 10 monclient(hunting): get_auth_request con 0x7f2b941421a0 auth_method 0 2023-12-07T23:46:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7f2b9a3dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7f2b9a3dc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7f2b9a3dc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 1 -- 172.21.15.129:0/3201794464 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c031740 con 0x7ff7a40391f0 2023-12-07T23:46:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: dump: 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.209 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 1 -- 172.21.15.129:0/3201794464 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c0318a0 con 0x7ff7a40391f0 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 1 -- 172.21.15.129:0/3201794464 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c031c70 con 0x7ff7a40391f0 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7a9ffb640 10 monclient: dump: 2023-12-07T23:46:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7b16b4640 5 monclient: authenticate success, global_id 4241 2023-12-07T23:46:16.211 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7b16b4640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.177+0000 7ff7b16b4640 4 set_mon_vals no callback set 2023-12-07T23:46:16.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7f2b993da640 10 monclient(hunting): get_auth_request con 0x7f2b940a9bd0 auth_method 0 2023-12-07T23:46:16.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7f2b993da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7f2b993da640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7f2b993da640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 10 monclient: shutdown 2023-12-07T23:46:16.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 -- 172.21.15.129:0/3201794464 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a40391f0 msgr2=0x7ff7a40c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 --2- 172.21.15.129:0/3201794464 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a40391f0 0x7ff7a40c85a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0097a0 tx=0x7ff79c02b7e0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 -- 172.21.15.129:0/3201794464 shutdown_connections 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 --2- 172.21.15.129:0/3201794464 >> v2:172.21.15.184:3300/0 conn(0x7ff7a4033d30 0x7ff7a40d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 --2- 172.21.15.129:0/3201794464 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a40391f0 0x7ff7a40c85a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 -- 172.21.15.129:0/3201794464 >> 172.21.15.129:0/3201794464 conn(0x7ff7a4006550 msgr2=0x7ff7a4006960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 -- 172.21.15.129:0/3201794464 shutdown_connections 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.178+0000 7ff7b16b4640 1 -- 172.21.15.129:0/3201794464 wait complete. 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.179+0000 7f7b7067c640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.179+0000 7f7b7067c640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.179+0000 7f7b7067c640 10 monclient: monmap: 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.180520+0000 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.180520+0000 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.216 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.217 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.217 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.217 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.217 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.217 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.217 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.179+0000 7ff7b16b4640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.217 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.179+0000 7ff7b16b4640 10 monclient: monmap: 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.218 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.179+0000 7f2b99bdb640 1 --2- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b940a8970 0x7f2b9413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7ff7b16b4640 1 librados: starting msgr at 2023-12-07T23:46:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7ff7b16b4640 1 librados: starting objecter 2023-12-07T23:46:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f7b7067c640 1 Processor -- start 2023-12-07T23:46:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f7b7067c640 1 -- start start 2023-12-07T23:46:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f2b99bdb640 10 monclient(hunting): get_auth_request con 0x7f2b940a8970 auth_method 0 2023-12-07T23:46:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f2b99bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f2b99bdb640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f2b99bdb640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f7b7067c640 10 monclient: init 2023-12-07T23:46:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7ff7b16b4640 1 Processor -- start 2023-12-07T23:46:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7ff7b16b4640 1 -- start start 2023-12-07T23:46:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7ff7b16b4640 1 librados: setting wanted keys 2023-12-07T23:46:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7ff7b16b4640 1 librados: calling monclient init 2023-12-07T23:46:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7ff7b16b4640 10 monclient: init 2023-12-07T23:46:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f2b9a3dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f2b9a3dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.180+0000 7f2b9a3dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f2b993da640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f2b993da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f2b993da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a4033d30 0x7ff7a40ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): picked mon.noname-c con 0x7ff7a4033d30 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7a4034630 0x7ff7a40ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 1 -- 172.21.15.129:0/2070174293 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50033640 con 0x7fcf54006c10 2023-12-07T23:46:16.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): picked mon.noname-a con 0x7ff7a4034630 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a40391f0 0x7ff7a415d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): picked mon.noname-b con 0x7ff7a40391f0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 10 monclient: dump: 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.225 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf5bfff640 5 monclient: authenticate success, global_id 4244 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf5bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40cb610 con 0x7ff7a4034630 2023-12-07T23:46:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40c9ba0 con 0x7ff7a40391f0 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40cb810 con 0x7ff7a4033d30 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7ff7b16b4640 10 monclient(hunting): authenticate will time out at 1376.369698s 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 1 -- 172.21.15.129:0/2070174293 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf500337a0 con 0x7fcf54006c10 2023-12-07T23:46:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf59ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf5bfff640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7fcf5bfff640 4 set_mon_vals no callback set 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7b680b1a70 0x7f7b680700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 10 monclient(hunting): picked mon.noname-a con 0x7f7b680b1a70 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b680710e0 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7b6813bcd0 0x7f7b6813c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6813bcd0 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.181+0000 7f7b7067c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6813c6a0 con 0x7f7b680710e0 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f7b7067c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f7b7067c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b680b1e50 con 0x7f7b680b1a70 2023-12-07T23:46:16.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f7b7067c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f7b7067c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b68070610 con 0x7f7b6813bcd0 2023-12-07T23:46:16.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f7b7067c640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f7b7067c640 10 monclient(hunting): authenticate will time out at 1376.370012s 2023-12-07T23:46:16.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf59ffb640 1 -- 172.21.15.129:0/2070174293 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50033b30 con 0x7fcf54006c10 2023-12-07T23:46:16.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf59ffb640 10 monclient: dump: 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.231 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.232 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fe74a7fc640 1 --2- 172.21.15.129:0/1880544313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744032020 0x7fe7440c4ea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe744008da0 tx=0x7fe74002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd82efba088970d server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f2b99bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f2b99bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7f2b99bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 10 monclient: shutdown 2023-12-07T23:46:16.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 -- 172.21.15.129:0/2070174293 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 msgr2=0x7fcf540cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 --2- 172.21.15.129:0/2070174293 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540cf5e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcf540076f0 tx=0x7fcf5002b720 comp rx=0 tx=0).stop 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 -- 172.21.15.129:0/2070174293 shutdown_connections 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 --2- 172.21.15.129:0/2070174293 >> v2:172.21.15.129:3300/0 conn(0x7fcf5402e630 0x7fcf54032880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 --2- 172.21.15.129:0/2070174293 >> v2:172.21.15.184:3300/0 conn(0x7fcf5402ea60 0x7fcf5402e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 --2- 172.21.15.129:0/2070174293 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540cf5e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 -- 172.21.15.129:0/2070174293 >> 172.21.15.129:0/2070174293 conn(0x7fcf54005e80 msgr2=0x7fcf54006290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 -- 172.21.15.129:0/2070174293 shutdown_connections 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.182+0000 7fcf5bfff640 1 -- 172.21.15.129:0/2070174293 wait complete. 2023-12-07T23:46:16.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.183+0000 7efd037fe640 1 --2- 172.21.15.129:0/201858483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd0413f470 0x7efd0406e640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009e50 tx=0x7efcfc02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b654f7722f537ed server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.183+0000 7fcf5bfff640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.183+0000 7fcf5bfff640 10 monclient: monmap: 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.235 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.183+0000 7efe77b7c640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.183+0000 7efe77b7c640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.183+0000 7efe77b7c640 10 monclient: monmap: 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.236 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.184951+0000 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.184951+0000 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.237 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 librados: starting msgr at 2023-12-07T23:46:16.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 librados: starting objecter 2023-12-07T23:46:16.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 Processor -- start 2023-12-07T23:46:16.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 -- start start 2023-12-07T23:46:16.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 librados: setting wanted keys 2023-12-07T23:46:16.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 librados: calling monclient init 2023-12-07T23:46:16.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient: init 2023-12-07T23:46:16.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7efe77b7c640 1 Processor -- start 2023-12-07T23:46:16.240 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.240 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:16.240 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.240 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fcf54006c10 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.240 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf5402e630 0x7fcf540cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.241 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7efe77b7c640 1 -- start start 2023-12-07T23:46:16.241 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7efe77b7c640 10 monclient: init 2023-12-07T23:46:16.241 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fcf5402e630 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.241 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf5402ea60 0x7fcf540cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.241 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fcf5402ea60 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540cef90 con 0x7fcf5402e630 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540cb3a0 con 0x7fcf5402ea60 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540ccc50 con 0x7fcf54006c10 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.184+0000 7fcf5bfff640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7fcf5bfff640 10 monclient(hunting): authenticate will time out at 1376.372980s 2023-12-07T23:46:16.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2023-12-07T23:46:16.243 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 1 -- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b940a8970 msgr2=0x7f2b9413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.243 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 1 --2- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b940a8970 0x7f2b9413ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.243 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 1 -- 172.21.15.129:0/2333347620 >> v2:172.21.15.184:3300/0 conn(0x7f2b940a9bd0 msgr2=0x7f2b9413f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.243 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7efe77b7c640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.243 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 1 --2- 172.21.15.129:0/2333347620 >> v2:172.21.15.184:3300/0 conn(0x7f2b940a9bd0 0x7f2b9413f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.243 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.243 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7f2b9a3dc640 1 -- 172.21.15.129:0/2333347620 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b940aaea0 con 0x7f2b941421a0 2023-12-07T23:46:16.245 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7efe77b7c640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-12-07T23:46:16.245 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7efe77b7c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efe70066510 0x7efe7013ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.245 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7efe77b7c640 10 monclient(hunting): picked mon.noname-c con 0x7efe70066510 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.245 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7efe77b7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe700afe40 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.245 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.185+0000 7efe77b7c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe701455a0 0x7efe70145a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 10 monclient(hunting): picked mon.noname-a con 0x7efe701455a0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe700afc10 con 0x7efe700afe40 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe700668f0 con 0x7efe701455a0 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe700af2d0 con 0x7efe70066510 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.186+0000 7efe77b7c640 10 monclient(hunting): authenticate will time out at 1376.374075s 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe749ffb640 1 -- 172.21.15.129:0/1880544313 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740031870 con 0x7fe744032020 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe749ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe749ffb640 10 monclient: dump: 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.247 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe749ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe749ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe7515a9640 5 monclient: authenticate success, global_id 4300 2023-12-07T23:46:16.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.187+0000 7fe7515a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:16.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 1 -- 172.21.15.129:0/4254535348 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4180317c0 con 0x7fe414033900 2023-12-07T23:46:16.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: dump: 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.250 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 1 -- 172.21.15.129:0/4254535348 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe418031920 con 0x7fe414033900 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 1 -- 172.21.15.129:0/4254535348 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe418031cb0 con 0x7fe414033900 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe4225fb640 5 monclient: authenticate success, global_id 4275 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe4225fb640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe4225fb640 4 set_mon_vals no callback set 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7fe413fff640 10 monclient: dump: 2023-12-07T23:46:16.251 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.252 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 1 -- 172.21.15.129:0/201858483 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc02c9f0 con 0x7efd0413f470 2023-12-07T23:46:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 10 monclient: dump: 2023-12-07T23:46:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd0968a640 5 monclient: authenticate success, global_id 4297 2023-12-07T23:46:16.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd0968a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:16.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 1 -- 172.21.15.129:0/201858483 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc02cb50 con 0x7efd0413f470 2023-12-07T23:46:16.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd02ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd0968a640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.188+0000 7efd0968a640 4 set_mon_vals no callback set 2023-12-07T23:46:16.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7f2b9a3dc640 1 --2- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b941421a0 0x7f2b94142630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2b940b4ea0 tx=0x7f2b8c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7aad40281e159cf server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd02ffd640 1 -- 172.21.15.129:0/201858483 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0343f0 con 0x7efd0413f470 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd02ffd640 10 monclient: dump: 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.256 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.257 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.257 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.257 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fcf5affd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 10 monclient: shutdown 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 -- 172.21.15.129:0/4254535348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 msgr2=0x7fe4140d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 --2- 172.21.15.129:0/4254535348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 0x7fe4140d24b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe418009870 tx=0x7fe41802b860 comp rx=0 tx=0).stop 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 -- 172.21.15.129:0/4254535348 shutdown_connections 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 --2- 172.21.15.129:0/4254535348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 0x7fe4140d24b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 --2- 172.21.15.129:0/4254535348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe414006fa0 0x7fe4140cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7ff7aaffd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a4033d30 0x7ff7a40ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 -- 172.21.15.129:0/4254535348 >> 172.21.15.129:0/4254535348 conn(0x7fe414006170 msgr2=0x7fe414006580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd0968a640 10 monclient: shutdown 2023-12-07T23:46:16.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd0968a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd0968a640 1 -- 172.21.15.129:0/201858483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd0413f470 msgr2=0x7efd0406e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.260 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd0968a640 1 --2- 172.21.15.129:0/201858483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd0413f470 0x7efd0406e640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009e50 tx=0x7efcfc02be50 comp rx=0 tx=0).stop 2023-12-07T23:46:16.260 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd0968a640 1 -- 172.21.15.129:0/201858483 shutdown_connections 2023-12-07T23:46:16.260 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 -- 172.21.15.129:0/4254535348 shutdown_connections 2023-12-07T23:46:16.260 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fe4225fb640 1 -- 172.21.15.129:0/4254535348 wait complete. 2023-12-07T23:46:16.260 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd0968a640 1 --2- 172.21.15.129:0/201858483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd0413f470 0x7efd0406e640 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.260 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7efd0968a640 1 -- 172.21.15.129:0/201858483 >> 172.21.15.129:0/201858483 conn(0x7efd040b0630 msgr2=0x7efd0406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.261 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fcf5affd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540ca860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55520/0 (socket says 172.21.15.129:55520) 2023-12-07T23:46:16.261 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fcf5affd640 1 -- 172.21.15.129:0/1536799848 learned_addr learned my addr 172.21.15.129:0/1536799848 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.261 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fcf5affd640 10 monclient(hunting): get_auth_request con 0x7fcf54006c10 auth_method 0 2023-12-07T23:46:16.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fcf5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.189+0000 7fcf5affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7fcf5affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7efd0968a640 1 -- 172.21.15.129:0/201858483 shutdown_connections 2023-12-07T23:46:16.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7efd0968a640 1 -- 172.21.15.129:0/201858483 wait complete. 2023-12-07T23:46:16.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7ff7aaffd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a4033d30 0x7ff7a40ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55516/0 (socket says 172.21.15.129:55516) 2023-12-07T23:46:16.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7ff7aaffd640 1 -- 172.21.15.129:0/1037897232 learned_addr learned my addr 172.21.15.129:0/1037897232 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7ff7aaffd640 10 monclient(hunting): get_auth_request con 0x7ff7a4033d30 auth_method 0 2023-12-07T23:46:16.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7ff7aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7ff7aaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7ff7aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7fe4225fb640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7fe4225fb640 10 monclient: monmap: 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.264 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.265 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.265 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.265 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.265 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.265 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7fcf5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.265 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7fcf5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.265 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7fcf5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7efd0968a640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7efd0968a640 10 monclient: monmap: 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.266 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.190+0000 7efd0968a640 1 librados: starting msgr at 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7efd0968a640 1 librados: starting objecter 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7fe4225fb640 1 librados: starting msgr at 2023-12-07T23:46:16.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7fe4225fb640 1 librados: starting objecter 2023-12-07T23:46:16.268 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.268 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.268 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.268 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7efd0968a640 1 Processor -- start 2023-12-07T23:46:16.268 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7fe4225fb640 1 Processor -- start 2023-12-07T23:46:16.268 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7efd0968a640 1 -- start start 2023-12-07T23:46:16.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7efd0968a640 1 librados: setting wanted keys 2023-12-07T23:46:16.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7efd0968a640 1 librados: calling monclient init 2023-12-07T23:46:16.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7fe4225fb640 1 -- start start 2023-12-07T23:46:16.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7fe4225fb640 1 librados: setting wanted keys 2023-12-07T23:46:16.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7fe4225fb640 1 librados: calling monclient init 2023-12-07T23:46:16.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7fe4225fb640 10 monclient: init 2023-12-07T23:46:16.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.191+0000 7efd0968a640 10 monclient: init 2023-12-07T23:46:16.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:16.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe414006fa0 0x7fe4140c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe414006fa0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 0x7fe4140c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe414033900 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4140c1870 0x7fe4140c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe4140c1870 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140c2260 con 0x7fe414033900 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140caee0 con 0x7fe414006fa0 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.272 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41415cc60 con 0x7fe4140c1870 2023-12-07T23:46:16.272 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.272 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7fe4225fb640 10 monclient(hunting): authenticate will time out at 1376.380217s 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd040aee70 0x7efd040b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): picked mon.noname-b con 0x7efd040aee70 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd0413f470 0x7efd04144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): picked mon.noname-a con 0x7efd0413f470 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd04145630 0x7efd040b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): picked mon.noname-c con 0x7efd04145630 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd041441f0 con 0x7efd0413f470 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd04130040 con 0x7efd040aee70 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd040b1d90 con 0x7efd04145630 2023-12-07T23:46:16.274 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.274 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.192+0000 7efd0968a640 10 monclient(hunting): authenticate will time out at 1376.380404s 2023-12-07T23:46:16.274 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.193+0000 7ff7abfff640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a40391f0 0x7ff7a415d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.274 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7f7b6ec19640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7b6813bcd0 0x7f7b6813c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.275 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2023-12-07T23:46:16.275 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 1 -- 172.21.15.129:0/1536799848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf5402ea60 msgr2=0x7fcf540cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.275 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf5402ea60 0x7fcf540cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.275 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 1 -- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf5402e630 msgr2=0x7fcf540cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.275 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf5402e630 0x7fcf540cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.275 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.275 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7fcf5affd640 1 -- 172.21.15.129:0/1536799848 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf50009400 con 0x7fcf54006c10 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7ff7abfff640 10 monclient(hunting): get_auth_request con 0x7ff7a40391f0 auth_method 0 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7ff7abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7ff7abfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.194+0000 7ff7abfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6ec19640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7b6813bcd0 0x7f7b6813c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40490/0 (socket says 172.21.15.129:40490) 2023-12-07T23:46:16.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6ec19640 1 -- 172.21.15.129:0/2290151946 learned_addr learned my addr 172.21.15.129:0/2290151946 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6ec19640 10 monclient(hunting): get_auth_request con 0x7f7b6813bcd0 auth_method 0 2023-12-07T23:46:16.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6ec19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6ec19640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6ec19640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6e418640 1 --2- 172.21.15.129:0/2290151946 >> v2:172.21.15.129:3300/0 conn(0x7f7b680b1a70 0x7f7b680700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7ff7aa7fc640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7a4034630 0x7ff7a40ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.278 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6e418640 10 monclient(hunting): get_auth_request con 0x7f7b680b1a70 auth_method 0 2023-12-07T23:46:16.278 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6e418640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.278 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6e418640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.278 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7f7b6e418640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7ff7aa7fc640 10 monclient(hunting): get_auth_request con 0x7ff7a4034630 auth_method 0 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7ff7aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7ff7aa7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.195+0000 7ff7aa7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7ff7abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7ff7abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7ff7abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7f7b6ec19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7f7b6ec19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7f7b6ec19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7ff7aa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7ff7aa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.196+0000 7ff7aa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.197+0000 7f7b6e418640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.197+0000 7f7b6e418640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.197+0000 7f7b6e418640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 1 -- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a40391f0 msgr2=0x7ff7a415d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a40391f0 0x7ff7a415d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 1 -- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7a4034630 msgr2=0x7ff7a40ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7a4034630 0x7ff7a40ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.198+0000 7ff7aaffd640 1 -- 172.21.15.129:0/1037897232 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c009400 con 0x7ff7a4033d30 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.200+0000 7fcf5affd640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540ca860 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009890 tx=0x7fcf500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.200+0000 7fcf437fe640 10 client.?.objecter ms_handle_connect 0x7fcf54006c10 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.200+0000 7fcf437fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.201+0000 7ff7aaffd640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a4033d30 0x7ff7a40ca030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff79c000c00 tx=0x7ff79c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.201+0000 7ff7937fe640 10 client.?.objecter ms_handle_connect 0x7ff7a4033d30 2023-12-07T23:46:16.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.201+0000 7ff7937fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 1 -- 172.21.15.129:0/2290151946 >> v2:172.21.15.129:3300/0 conn(0x7f7b680b1a70 msgr2=0x7f7b680700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 1 --2- 172.21.15.129:0/2290151946 >> v2:172.21.15.129:3300/0 conn(0x7f7b680b1a70 0x7f7b680700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 1 -- 172.21.15.129:0/2290151946 >> v1:172.21.15.184:6789/0 conn(0x7f7b680710e0 legacy=0x7f7b680b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.284 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.202+0000 7f7b6ec19640 1 -- 172.21.15.129:0/2290151946 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68072420 con 0x7f7b6813bcd0 2023-12-07T23:46:16.285 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.204+0000 7fe4215f9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe414006fa0 0x7fe4140c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.285 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7efe76119640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe701455a0 0x7efe70145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.285 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7efd03fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd040aee70 0x7efd040b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe749ffb640 1 -- 172.21.15.129:0/1880544313 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7400319d0 con 0x7fe744032020 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe749ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe749ffb640 1 -- 172.21.15.129:0/1880544313 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740031d60 con 0x7fe744032020 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe749ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe749ffb640 10 monclient: dump: 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.286 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe7515a9640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.287 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe7515a9640 4 set_mon_vals no callback set 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 1 -- 172.21.15.129:0/2333347620 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c02c960 con 0x7f2b941421a0 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: dump: 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.288 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe4215f9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe414006fa0 0x7fe4140c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40524/0 (socket says 172.21.15.129:40524) 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe4215f9640 1 -- 172.21.15.129:0/175273291 learned_addr learned my addr 172.21.15.129:0/175273291 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe4215f9640 10 monclient(hunting): get_auth_request con 0x7fe414006fa0 auth_method 0 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe4215f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe4215f9640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.290 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7fe4215f9640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.290 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 1 -- 172.21.15.129:0/2333347620 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c02cac0 con 0x7f2b941421a0 2023-12-07T23:46:16.290 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.290 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 1 -- 172.21.15.129:0/2333347620 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c02ce50 con 0x7f2b941421a0 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b98bd9640 10 monclient: dump: 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b9be3f640 5 monclient: authenticate success, global_id 4281 2023-12-07T23:46:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b9be3f640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.292 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.205+0000 7f2b9be3f640 4 set_mon_vals no callback set 2023-12-07T23:46:16.292 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7efe75918640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efe70066510 0x7efe7013ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50030980 con 0x7fcf54006c10 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: dump: 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf50030b00 con 0x7fcf54006c10 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50030eb0 con 0x7fcf54006c10 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf437fe640 10 monclient: dump: 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7f2b9be3f640 10 monclient: shutdown 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 10 monclient: shutdown 2023-12-07T23:46:16.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 1 -- 172.21.15.129:0/1880544313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744032020 msgr2=0x7fe7440c4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 1 --2- 172.21.15.129:0/1880544313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744032020 0x7fe7440c4ea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe744008da0 tx=0x7fe74002b910 comp rx=0 tx=0).stop 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7f2b9be3f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b941421a0 msgr2=0x7f2b94142630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7efd03fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd040aee70 0x7efd040b0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40534/0 (socket says 172.21.15.129:40534) 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7efd03fff640 1 -- 172.21.15.129:0/1232236853 learned_addr learned my addr 172.21.15.129:0/1232236853 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7efd03fff640 10 monclient(hunting): get_auth_request con 0x7efd040aee70 auth_method 0 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7efd03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7efd03fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.297 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7efd03fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.297 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7f2b9be3f640 1 --2- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b941421a0 0x7f2b94142630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2b940b4ea0 tx=0x7f2b8c02bdc0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.297 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 1 -- 172.21.15.129:0/1880544313 shutdown_connections 2023-12-07T23:46:16.297 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 1 --2- 172.21.15.129:0/1880544313 >> v2:172.21.15.129:3300/0 conn(0x7fe7440c53e0 0x7fe7440d24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.297 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 1 --2- 172.21.15.129:0/1880544313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744032020 0x7fe7440c4ea0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.297 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fe7515a9640 1 -- 172.21.15.129:0/1880544313 >> 172.21.15.129:0/1880544313 conn(0x7fe744006a20 msgr2=0x7fe744006e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2333347620 shutdown_connections 2023-12-07T23:46:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7f2b9be3f640 1 --2- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b941421a0 0x7f2b94142630 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7f2b9be3f640 1 --2- 172.21.15.129:0/2333347620 >> v2:172.21.15.184:3300/0 conn(0x7f2b940a9bd0 0x7f2b9413f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7f2b9be3f640 1 --2- 172.21.15.129:0/2333347620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b940a8970 0x7f2b9413ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2333347620 >> 172.21.15.129:0/2333347620 conn(0x7f2b94066dd0 msgr2=0x7f2b940b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7fe7515a9640 1 -- 172.21.15.129:0/1880544313 shutdown_connections 2023-12-07T23:46:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7efd037fe640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd0413f470 0x7efd04144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7fe7515a9640 1 -- 172.21.15.129:0/1880544313 wait complete. 2023-12-07T23:46:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2333347620 shutdown_connections 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2333347620 wait complete. 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 5 monclient: authenticate success, global_id 4250 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 10 monclient: _renew_subs 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf540cc970 con 0x7fcf54006c10 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf597fa640 4 set_mon_vals no callback set 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 10 monclient: _renew_subs 2023-12-07T23:46:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf540cb830 con 0x7fcf54006c10 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.206+0000 7fcf5bfff640 1 librados: init done 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.207+0000 7fe420df8640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 0x7fe4140c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 10 monclient: monmap: 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fcf5bfff640 10 client.4250.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fcf5bfff640 15 client.4250.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.302 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fcf5bfff640 20 client.4250.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe4215f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe4215f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe4215f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fcf5bfff640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fcf5bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf540cb4e0 con 0x7fcf54006c10 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 10 monclient: monmap: 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.304 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efd03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efd03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efd03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe76119640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe701455a0 0x7efe70145a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42814/0 (socket says 172.21.15.129:42814) 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe76119640 1 -- 172.21.15.129:0/3846305257 learned_addr learned my addr 172.21.15.129:0/3846305257 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe76119640 10 monclient(hunting): get_auth_request con 0x7efe701455a0 auth_method 0 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe76119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe76119640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.305 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe76119640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 1 librados: starting msgr at 2023-12-07T23:46:16.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 1 librados: starting objecter 2023-12-07T23:46:16.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 1 librados: starting msgr at 2023-12-07T23:46:16.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 1 librados: starting objecter 2023-12-07T23:46:16.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 1 Processor -- start 2023-12-07T23:46:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe420df8640 10 monclient(hunting): get_auth_request con 0x7fe414033900 auth_method 0 2023-12-07T23:46:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe420df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe420df8640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe420df8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 1 -- start start 2023-12-07T23:46:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe75918640 10 monclient(hunting): get_auth_request con 0x7efe70066510 auth_method 0 2023-12-07T23:46:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe75918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe75918640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efe75918640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 1 librados: setting wanted keys 2023-12-07T23:46:16.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 1 librados: calling monclient init 2023-12-07T23:46:16.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7f2b9be3f640 10 monclient: init 2023-12-07T23:46:16.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 1 Processor -- start 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efd037fe640 10 monclient(hunting): get_auth_request con 0x7efd0413f470 auth_method 0 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efd037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efd037fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7efd037fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 1 -- start start 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 1 librados: setting wanted keys 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 1 librados: calling monclient init 2023-12-07T23:46:16.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.208+0000 7fe7515a9640 10 monclient: init 2023-12-07T23:46:16.310 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.310 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:16.310 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b940a8970 0x7f2b9413df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): picked mon.noname-b con 0x7f2b940a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.313 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b940a9bd0 0x7f2b940b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.313 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b940a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.313 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b941421a0 0x7f2b940b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.313 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7efd037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7fe7515a9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b941421a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7efd037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7fe7515a9640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:46:16.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7fe7515a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744006400 0x7fe744038f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b940b5ef0 con 0x7f2b940a9bd0 2023-12-07T23:46:16.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7efd037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b940b05e0 con 0x7f2b940a8970 2023-12-07T23:46:16.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7fe7515a9640 10 monclient(hunting): picked mon.noname-b con 0x7fe744006400 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b940b07a0 con 0x7f2b941421a0 2023-12-07T23:46:16.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7f2b9be3f640 10 monclient(hunting): authenticate will time out at 1376.397731s 2023-12-07T23:46:16.317 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7fe7515a9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe744032020 0x7fe744160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.317 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7fe7515a9640 10 monclient(hunting): picked mon.noname-c con 0x7fe744032020 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.317 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.209+0000 7fe7515a9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7440c53e0 0x7fe744160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.317 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 10 monclient(hunting): picked mon.noname-a con 0x7fe7440c53e0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.317 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74402c7f0 con 0x7fe7440c53e0 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440ca2c0 con 0x7fe744006400 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440ca530 con 0x7fe744032020 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7fe7515a9640 10 monclient(hunting): authenticate will time out at 1376.398018s 2023-12-07T23:46:16.318 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c033980 con 0x7ff7a4033d30 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: dump: 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c033b00 con 0x7ff7a4033d30 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c03d410 con 0x7ff7a4033d30 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7937fe640 10 monclient: dump: 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 5 monclient: authenticate success, global_id 4253 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 10 monclient: _renew_subs 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a40cbaf0 con 0x7ff7a4033d30 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7a97fa640 4 set_mon_vals no callback set 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 10 monclient: _renew_subs 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a40cae40 con 0x7ff7a4033d30 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7ff7b16b4640 1 librados: init done 2023-12-07T23:46:16.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7efe76119640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7efe76119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.210+0000 7efe76119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 1 -- 172.21.15.129:0/1232236853 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd04145630 msgr2=0x7efd040b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd04145630 0x7efd040b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 1 -- 172.21.15.129:0/1232236853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd0413f470 msgr2=0x7efd04144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd0413f470 0x7efd04144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7efd03fff640 1 -- 172.21.15.129:0/1232236853 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc009b30 con 0x7efd040aee70 2023-12-07T23:46:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7ff7b16b4640 10 client.4253.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7ff7b16b4640 15 client.4253.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7ff7b16b4640 20 client.4253.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7ff7b16b4640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7ff7b16b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.211+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff7a4039de0 con 0x7ff7a4033d30 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 1 -- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4140c1870 msgr2=0x7fe4140c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4140c1870 0x7fe4140c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 1 -- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 msgr2=0x7fe4140c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe414033900 0x7fe4140c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.213+0000 7fe4215f9640 1 -- 172.21.15.129:0/175273291 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4180094d0 con 0x7fe414006fa0 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.214+0000 7efe75918640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.214+0000 7efe75918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.214+0000 7efe75918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 1 -- 172.21.15.129:0/3846305257 >> v2:172.21.15.184:3300/0 conn(0x7efe70066510 msgr2=0x7efe7013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 1 --2- 172.21.15.129:0/3846305257 >> v2:172.21.15.184:3300/0 conn(0x7efe70066510 0x7efe7013ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 1 -- 172.21.15.129:0/3846305257 >> v1:172.21.15.184:6789/0 conn(0x7efe700afe40 legacy=0x7efe7013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.217+0000 7efe76119640 1 -- 172.21.15.129:0/3846305257 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe700ae6c0 con 0x7efe701455a0 2023-12-07T23:46:16.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.218+0000 7fe4215f9640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe414006fa0 0x7fe4140c8d40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe41c064260 tx=0x7fe41c0678c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.219+0000 7fe411ffb640 10 client.?.objecter ms_handle_connect 0x7fe414006fa0 2023-12-07T23:46:16.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.219+0000 7fe411ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.219+0000 7efd03fff640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd040aee70 0x7efd040b0de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c460 tx=0x7efcf800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.219+0000 7efd00ff9640 10 client.?.objecter ms_handle_connect 0x7efd040aee70 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.220+0000 7efd00ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.220+0000 7f2b993da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b940a9bd0 0x7f2b940b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.221+0000 7f2b993da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b940a9bd0 0x7f2b940b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42830/0 (socket says 172.21.15.129:42830) 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.221+0000 7f2b993da640 1 -- 172.21.15.129:0/2010684625 learned_addr learned my addr 172.21.15.129:0/2010684625 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.221+0000 7f2b993da640 10 monclient(hunting): get_auth_request con 0x7f2b940a9bd0 auth_method 0 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.221+0000 7f2b993da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.221+0000 7f2b993da640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.221+0000 7f2b993da640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.221+0000 7efe76119640 1 --2- 172.21.15.129:0/3846305257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe70145a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efe700ad8c0 tx=0x7efe6802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d4ada6c659096e server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.222+0000 7f2b993da640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.222+0000 7f2b993da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.222+0000 7f2b993da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe74affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744006400 0x7fe744038f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 1 -- 172.21.15.129:0/2010684625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b941421a0 msgr2=0x7f2b940b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b941421a0 0x7f2b940b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 1 -- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b940a8970 msgr2=0x7f2b9413df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b940a8970 0x7f2b9413df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7f2b993da640 1 -- 172.21.15.129:0/2010684625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c009b30 con 0x7f2b940a9bd0 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c06cba0 con 0x7fe414006fa0 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: dump: 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.331 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c06cd00 con 0x7fe414006fa0 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe4225fb640 5 monclient: authenticate success, global_id 4312 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c093470 con 0x7fe414006fa0 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe4225fb640 10 monclient: _renew_subs 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe4225fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4140cb130 con 0x7fe414006fa0 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe4137fe640 4 set_mon_vals no callback set 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.228+0000 7fe411ffb640 10 monclient: dump: 2023-12-07T23:46:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 10 monclient: _renew_subs 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4140c60c0 con 0x7fe414006fa0 2023-12-07T23:46:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 1 librados: init done 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe74affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744006400 0x7fe744038f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40562/0 (socket says 172.21.15.129:40562) 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe74affd640 1 -- 172.21.15.129:0/2131958372 learned_addr learned my addr 172.21.15.129:0/2131958372 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe74affd640 10 monclient(hunting): get_auth_request con 0x7fe744006400 auth_method 0 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe74affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe74affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe74affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 10 client.4312.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 15 client.4312.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 20 client.4312.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.229+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe4140375e0 con 0x7fe414006fa0 2023-12-07T23:46:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.231+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcf5003a050 con 0x7fcf54006c10 2023-12-07T23:46:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.232+0000 7f7b6ec19640 1 --2- 172.21.15.129:0/2290151946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813bcd0 0x7f7b6813c160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7b680aebd0 tx=0x7f7b60030260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b6299b91f60734 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.232+0000 7fcf437fe640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fcf4806c870 0x7fcf4806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.232+0000 7fcf5a7fc640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fcf4806c870 0x7fcf4806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.232+0000 7fcf5a7fc640 10 monclient: get_auth_request con 0x7fcf4806c870 auth_method 0 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.232+0000 7fe74affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.232+0000 7fe74affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.232+0000 7fe74affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.233+0000 7fcf5a7fc640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fcf4806c870 0x7fcf4806ed30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cf290 tx=0x7fcf4c009500 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.233+0000 7fcf437fe640 10 client.4250.objecter ms_handle_connect 0x7fcf4806c870 2023-12-07T23:46:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.233+0000 7f2b993da640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b940a9bd0 0x7f2b940b52c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e980 tx=0x7f2b8400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.233+0000 7f2b8a7fc640 10 client.?.objecter ms_handle_connect 0x7f2b940a9bd0 2023-12-07T23:46:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.233+0000 7f2b8a7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 1 -- 172.21.15.129:0/3846305257 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6802ca80 con 0x7efe701455a0 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: dump: 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 1 -- 172.21.15.129:0/3846305257 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6802cbe0 con 0x7efe701455a0 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 1 -- 172.21.15.129:0/3846305257 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe680343f0 con 0x7efe701455a0 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe74916640 10 monclient: dump: 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 5 monclient: authenticate success, global_id 4293 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 4 set_mon_vals no callback set 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 10 monclient: shutdown 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 1 -- 172.21.15.129:0/3846305257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 msgr2=0x7efe70145a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 1 --2- 172.21.15.129:0/3846305257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe70145a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efe700ad8c0 tx=0x7efe6802bee0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 1 -- 172.21.15.129:0/3846305257 shutdown_connections 2023-12-07T23:46:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 1 --2- 172.21.15.129:0/3846305257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe70145a30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 1 --2- 172.21.15.129:0/3846305257 >> v2:172.21.15.184:3300/0 conn(0x7efe70066510 0x7efe7013ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.235+0000 7efe77b7c640 1 -- 172.21.15.129:0/3846305257 >> 172.21.15.129:0/3846305257 conn(0x7efe700b3ea0 msgr2=0x7efe700b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.236+0000 7efe77b7c640 1 -- 172.21.15.129:0/3846305257 shutdown_connections 2023-12-07T23:46:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.236+0000 7efe77b7c640 1 -- 172.21.15.129:0/3846305257 wait complete. 2023-12-07T23:46:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.236+0000 7fe74b7fe640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7440c53e0 0x7fe744160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.236+0000 7efe77b7c640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.236+0000 7efe77b7c640 10 monclient: monmap: 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.342 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcf5007b120 con 0x7fcf54006c10 2023-12-07T23:46:16.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 10 client.4250.objecter ms_dispatch 0x7fcf5c000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 3 client.4250.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 3 client.4250.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 librados: starting msgr at 2023-12-07T23:46:16.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 librados: starting objecter 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 10 client.4250.objecter checking command 1 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7fcf48072c50 0x7fcf48075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 20 client.4250.objecter _get_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 20 client.4250.objecter put_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 15 client.4250.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 20 client.4250.objecter _get_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 20 client.4250.objecter get_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 15 client.4250.objecter _session_command_op_assign 3 1 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 20 client.4250.objecter put_session s=0x7fcf48072580 osd=3 4 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 10 client.4250.objecter _send_command 1 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 --> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf48075720 con 0x7fcf48072c50 2023-12-07T23:46:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 20 client.4250.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf50076d60 con 0x7fcf54006c10 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf437fe640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 Processor -- start 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 -- start start 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 librados: setting wanted keys 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 librados: calling monclient init 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 10 monclient: init 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T23:46:16.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fcf5b7fe640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7fcf48072c50 0x7fcf48075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fe74b7fe640 10 monclient(hunting): get_auth_request con 0x7fe7440c53e0 auth_method 0 2023-12-07T23:46:16.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fe74b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fe74b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7fe74b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe70066510 0x7efe70142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 10 monclient(hunting): picked mon.noname-c con 0x7efe70066510 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe700afe40 0x7efe701d7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 10 monclient(hunting): picked mon.noname-b con 0x7efe700afe40 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe701d7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 10 monclient(hunting): picked mon.noname-a con 0x7efe701455a0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.237+0000 7efe77b7c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7011cdc0 con 0x7efe701455a0 2023-12-07T23:46:16.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7efe77b7c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7efe77b7c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7011cfe0 con 0x7efe700afe40 2023-12-07T23:46:16.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7efe77b7c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7efe77b7c640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe70142e20 con 0x7efe70066510 2023-12-07T23:46:16.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7efe77b7c640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7efe77b7c640 10 monclient(hunting): authenticate will time out at 1376.426000s 2023-12-07T23:46:16.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7fcf5b7fe640 10 monclient: get_auth_request con 0x7fcf48072c50 auth_method 0 2023-12-07T23:46:16.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.238+0000 7fe74b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.239+0000 7fe74b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.239+0000 7fe74b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.239+0000 7fe74a7fc640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe744032020 0x7fe744160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fcf437fe640 10 client.4250.objecter ms_handle_connect 0x7fcf48072c50 2023-12-07T23:46:16.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fcf5b7fe640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7fcf48072c50 0x7fcf48075070 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2023-12-07T23:46:16.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 1 -- 172.21.15.129:0/2131958372 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe744032020 msgr2=0x7fe744160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe744032020 0x7fe744160590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 1 -- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744006400 msgr2=0x7fe744038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe744006400 0x7fe744038f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.242+0000 7fe74b7fe640 1 -- 172.21.15.129:0/2131958372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400094d0 con 0x7fe7440c53e0 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.245+0000 7fe74b7fe640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7440c53e0 0x7fe744160ad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe74c068930 tx=0x7fe74c0678c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.245+0000 7fe7337fe640 10 client.?.objecter ms_handle_connect 0x7fe7440c53e0 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.245+0000 7fe7337fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.246+0000 7efe76119640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe701d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.246+0000 7efe76119640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe701d7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42848/0 (socket says 172.21.15.129:42848) 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.246+0000 7efe76119640 1 -- 172.21.15.129:0/1903207237 learned_addr learned my addr 172.21.15.129:0/1903207237 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.246+0000 7efe76119640 10 monclient(hunting): get_auth_request con 0x7efe701455a0 auth_method 0 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.246+0000 7efe76119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.246+0000 7efe76119640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.246+0000 7efe76119640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.249+0000 7efe76119640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.249+0000 7efe76119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.249+0000 7efe76119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.250+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84009670 con 0x7f2b940a9bd0 2023-12-07T23:46:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.250+0000 7f2b8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: dump: 2023-12-07T23:46:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8400ca00 con 0x7f2b940a9bd0 2023-12-07T23:46:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8400cdb0 con 0x7f2b940a9bd0 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8a7fc640 10 monclient: dump: 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b8bfff640 4 set_mon_vals no callback set 2023-12-07T23:46:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 5 monclient: authenticate success, global_id 4299 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 10 monclient: _renew_subs 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b940b09c0 con 0x7f2b940a9bd0 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 10 monclient: _renew_subs 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b940b0e80 con 0x7f2b940a9bd0 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.251+0000 7f2b9be3f640 1 librados: init done 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.252+0000 7f2b9be3f640 10 client.4299.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.252+0000 7f2b9be3f640 15 client.4299.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.252+0000 7f2b9be3f640 20 client.4299.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.252+0000 7f2b9be3f640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.252+0000 7f2b9be3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.253+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2b940b6060 con 0x7f2b940a9bd0 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.253+0000 7efe76119640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 1 -- 172.21.15.129:0/1903207237 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe70066510 msgr2=0x7efe70142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe70066510 0x7efe70142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 1 -- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe700afe40 msgr2=0x7efe701d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe700afe40 0x7efe701d7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.254+0000 7efe76119640 1 -- 172.21.15.129:0/1903207237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe68009b30 con 0x7efe701455a0 2023-12-07T23:46:16.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c06cba0 con 0x7fe7440c53e0 2023-12-07T23:46:16.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 10 monclient: dump: 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c072050 con 0x7fe7440c53e0 2023-12-07T23:46:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7337fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.255+0000 7fe7497fa640 4 set_mon_vals no callback set 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c0721c0 con 0x7fe7440c53e0 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 5 monclient: authenticate success, global_id 4302 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 10 monclient: _renew_subs 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74402b970 con 0x7fe7440c53e0 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7337fe640 10 monclient: dump: 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 10 monclient: _renew_subs 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7440bdd70 con 0x7fe7440c53e0 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.256+0000 7fe7515a9640 1 librados: init done 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.257+0000 7fe7515a9640 10 client.4302.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.257+0000 7fe7515a9640 15 client.4302.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.257+0000 7fe7515a9640 20 client.4302.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.257+0000 7fe7515a9640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.257+0000 7fe7515a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.257+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe74402c680 con 0x7fe7440c53e0 2023-12-07T23:46:16.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.259+0000 7efe76119640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe701d7550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe68036040 tx=0x7efe68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.259+0000 7efe667fc640 10 client.?.objecter ms_handle_connect 0x7efe701455a0 2023-12-07T23:46:16.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.259+0000 7efe667fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.259+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe41c090070 con 0x7fe414006fa0 2023-12-07T23:46:16.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.260+0000 7fe411ffb640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe40006c8a0 0x7fe40006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.260+0000 7fe420df8640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe40006c8a0 0x7fe40006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.260+0000 7fe420df8640 10 monclient: get_auth_request con 0x7fe40006c8a0 auth_method 0 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.260+0000 7fe411ffb640 10 client.4312.objecter ms_handle_connect 0x7fe40006c8a0 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.260+0000 7fe420df8640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe40006c8a0 0x7fe40006ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe40c007fd0 tx=0x7fe40c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe41c0d2960 con 0x7fe414006fa0 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 10 client.4312.objecter ms_dispatch 0x7fe41c000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 3 client.4312.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 3 client.4312.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 10 client.4312.objecter checking command 1 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7fe400072c80 0x7fe4000750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 20 client.4312.objecter _get_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 20 client.4312.objecter put_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 15 client.4312.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 20 client.4312.objecter _get_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 20 client.4312.objecter get_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 15 client.4312.objecter _session_command_op_assign 0 1 2023-12-07T23:46:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 20 client.4312.objecter put_session s=0x7fe4000725b0 osd=0 4 2023-12-07T23:46:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 10 client.4312.objecter _send_command 1 2023-12-07T23:46:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 --> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe400075750 con 0x7fe400072c80 2023-12-07T23:46:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 20 client.4312.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe41c0ce5a0 con 0x7fe414006fa0 2023-12-07T23:46:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe411ffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.262+0000 7fe421dfa640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7fe400072c80 0x7fe4000750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.263+0000 7fe421dfa640 10 monclient: get_auth_request con 0x7fe400072c80 auth_method 0 2023-12-07T23:46:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8012830 con 0x7efd040aee70 2023-12-07T23:46:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: dump: 2023-12-07T23:46:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf80129d0 con 0x7efd040aee70 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 5 monclient: authenticate success, global_id 4315 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 10 monclient: _renew_subs 2023-12-07T23:46:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd040b2340 con 0x7efd040aee70 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8012d80 con 0x7efd040aee70 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd00ff9640 10 monclient: dump: 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.369 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd027fc640 4 set_mon_vals no callback set 2023-12-07T23:46:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 10 monclient: _renew_subs 2023-12-07T23:46:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd041447f0 con 0x7efd040aee70 2023-12-07T23:46:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.264+0000 7efd0968a640 1 librados: init done 2023-12-07T23:46:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 1 -- 172.21.15.129:0/2290151946 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60030e00 con 0x7f7b6813bcd0 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: dump: 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 1 -- 172.21.15.129:0/2290151946 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b600383f0 con 0x7f7b6813bcd0 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 1 -- 172.21.15.129:0/2290151946 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60038780 con 0x7f7b6813bcd0 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b6d416640 10 monclient: dump: 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.373 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b7067c640 5 monclient: authenticate success, global_id 4309 2023-12-07T23:46:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b7067c640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7f7b7067c640 4 set_mon_vals no callback set 2023-12-07T23:46:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7efd0968a640 10 client.4315.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7efd0968a640 15 client.4315.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7efd0968a640 20 client.4315.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7efd0968a640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7efd0968a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.265+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd0406ecb0 con 0x7efd040aee70 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 10 monclient: shutdown 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 -- 172.21.15.129:0/2290151946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813bcd0 msgr2=0x7f7b6813c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 --2- 172.21.15.129:0/2290151946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813bcd0 0x7f7b6813c160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7b680aebd0 tx=0x7f7b60030260 comp rx=0 tx=0).stop 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 -- 172.21.15.129:0/2290151946 shutdown_connections 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 --2- 172.21.15.129:0/2290151946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813bcd0 0x7f7b6813c160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 --2- 172.21.15.129:0/2290151946 >> v2:172.21.15.129:3300/0 conn(0x7f7b680b1a70 0x7f7b680700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 -- 172.21.15.129:0/2290151946 >> 172.21.15.129:0/2290151946 conn(0x7f7b680b4c20 msgr2=0x7f7b680b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 -- 172.21.15.129:0/2290151946 shutdown_connections 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.266+0000 7f7b7067c640 1 -- 172.21.15.129:0/2290151946 wait complete. 2023-12-07T23:46:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 10 monclient: monmap: 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7fe421dfa640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7fe400072c80 0x7fe4000750a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7fe411ffb640 10 client.4312.objecter ms_handle_connect 0x7fe400072c80 2023-12-07T23:46:16.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 1 librados: starting msgr at 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 1 librados: starting objecter 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 1 Processor -- start 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 1 -- start start 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 1 librados: setting wanted keys 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 1 librados: calling monclient init 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.267+0000 7f7b7067c640 10 monclient: init 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680710e0 0x7f7b680b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): picked mon.noname-c con 0x7f7b680710e0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b680b1a70 0x7f7b68141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): picked mon.noname-a con 0x7f7b680b1a70 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813bcd0 0x7f7b681d5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6813bcd0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6813d6c0 con 0x7f7b680b1a70 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b68142570 con 0x7f7b6813bcd0 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b68142a90 con 0x7f7b680710e0 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.268+0000 7f7b7067c640 10 monclient(hunting): authenticate will time out at 1376.456428s 2023-12-07T23:46:16.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.270+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff79c03a050 con 0x7ff7a4033d30 2023-12-07T23:46:16.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.270+0000 7ff7937fe640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ff79806c870 0x7ff79806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.272+0000 7ff7aa7fc640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ff79806c870 0x7ff79806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.272+0000 7ff7aa7fc640 10 monclient: get_auth_request con 0x7ff79806c870 auth_method 0 2023-12-07T23:46:16.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.272+0000 7ff7937fe640 10 client.4253.objecter ms_handle_connect 0x7ff79806c870 2023-12-07T23:46:16.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.272+0000 7ff7aa7fc640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ff79806c870 0x7ff79806ed30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac02cee0 tx=0x7ff7ac068400 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff79c07a570 con 0x7ff7a4033d30 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 10 client.4253.objecter ms_dispatch 0x7ff7ac000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 3 client.4253.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 3 client.4253.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 10 client.4253.objecter checking command 1 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7ff798072c50 0x7ff798075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 20 client.4253.objecter _get_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 20 client.4253.objecter put_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 15 client.4253.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 20 client.4253.objecter _get_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 20 client.4253.objecter get_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 15 client.4253.objecter _session_command_op_assign 1 1 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 20 client.4253.objecter put_session s=0x7ff798072580 osd=1 4 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 10 client.4253.objecter _send_command 1 2023-12-07T23:46:16.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff798075720 con 0x7ff798072c50 2023-12-07T23:46:16.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 20 client.4253.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff79c0768b0 con 0x7ff7a4033d30 2023-12-07T23:46:16.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7937fe640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.274+0000 7ff7abfff640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7ff798072c50 0x7ff798075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.275+0000 7ff7abfff640 10 monclient: get_auth_request con 0x7ff798072c50 auth_method 0 2023-12-07T23:46:16.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7ff7937fe640 10 client.4253.objecter ms_handle_connect 0x7ff798072c50 2023-12-07T23:46:16.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7ff7abfff640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7ff798072c50 0x7ff798075070 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7f7b6e418640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680710e0 0x7f7b680b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7f7b6e418640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680710e0 0x7f7b680b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55576/0 (socket says 172.21.15.129:55576) 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7f7b6e418640 1 -- 172.21.15.129:0/1597030079 learned_addr learned my addr 172.21.15.129:0/1597030079 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7f7b6e418640 10 monclient(hunting): get_auth_request con 0x7f7b680710e0 auth_method 0 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7f7b6e418640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7f7b6e418640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.278+0000 7f7b6e418640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.279+0000 7f7b6e418640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.279+0000 7f7b6e418640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.279+0000 7f7b6e418640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 1 -- 172.21.15.129:0/1597030079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813bcd0 msgr2=0x7f7b681d5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b6813bcd0 0x7f7b681d5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 1 -- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b680b1a70 msgr2=0x7f7b68141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b680b1a70 0x7f7b68141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.283+0000 7f7b6e418640 1 -- 172.21.15.129:0/1597030079 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b60009b30 con 0x7f7b680710e0 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.285+0000 7f7b6e418640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680710e0 0x7f7b680b3670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400e9b0 tx=0x7f7b6400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.285+0000 7f7b5effd640 10 client.?.objecter ms_handle_connect 0x7f7b680710e0 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.285+0000 7f7b5effd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64009670 con 0x7f7b680710e0 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: dump: 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.385 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6400c9a0 con 0x7f7b680710e0 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6400cd50 con 0x7f7b680710e0 2023-12-07T23:46:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b5effd640 10 monclient: dump: 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 5 monclient: authenticate success, global_id 4256 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.288+0000 7f7b6cc15640 4 set_mon_vals no callback set 2023-12-07T23:46:16.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 10 monclient: _renew_subs 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b681420c0 con 0x7f7b680710e0 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efcf8024430 con 0x7efd040aee70 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 10 monclient: _renew_subs 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b68142340 con 0x7f7b680710e0 2023-12-07T23:46:16.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7f7b7067c640 1 librados: init done 2023-12-07T23:46:16.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.289+0000 7efd00ff9640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efcf006c870 0x7efcf006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7efd037fe640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efcf006c870 0x7efcf006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7efd037fe640 10 monclient: get_auth_request con 0x7efcf006c870 auth_method 0 2023-12-07T23:46:16.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7f7b7067c640 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7f7b7067c640 15 client.4256.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7f7b7067c640 20 client.4256.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7f7b7067c640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7f7b7067c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7efd00ff9640 10 client.4315.objecter ms_handle_connect 0x7efcf006c870 2023-12-07T23:46:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7efd037fe640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efcf006c870 0x7efcf006ed30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efcfc035000 tx=0x7efcfc030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.290+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7b6806a4c0 con 0x7f7b680710e0 2023-12-07T23:46:16.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efcf8058d80 con 0x7efd040aee70 2023-12-07T23:46:16.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 10 client.4315.objecter ms_dispatch 0x7efd04000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 3 client.4315.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 3 client.4315.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 10 client.4315.objecter checking command 1 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7efcf0072c50 0x7efcf0075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 20 client.4315.objecter _get_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 20 client.4315.objecter put_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 15 client.4315.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 20 client.4315.objecter _get_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 20 client.4315.objecter get_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 15 client.4315.objecter _session_command_op_assign 5 1 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 20 client.4315.objecter put_session s=0x7efcf0072580 osd=5 4 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 10 client.4315.objecter _send_command 1 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 --> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efcf0075720 con 0x7efcf0072c50 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 20 client.4315.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcf80549c0 con 0x7efd040aee70 2023-12-07T23:46:16.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.291+0000 7efd00ff9640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.293+0000 7efd08e89640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7efcf0072c50 0x7efcf0075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.293+0000 7efd08e89640 10 monclient: get_auth_request con 0x7efcf0072c50 auth_method 0 2023-12-07T23:46:16.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.296+0000 7efd00ff9640 10 client.4315.objecter ms_handle_connect 0x7efcf0072c50 2023-12-07T23:46:16.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.297+0000 7efd08e89640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7efcf0072c50 0x7efcf0075070 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.305+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2b84013470 con 0x7f2b940a9bd0 2023-12-07T23:46:16.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.305+0000 7f2b8a7fc640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f2b7806c820 0x7f2b7806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.305+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe68009510 con 0x7efe701455a0 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.305+0000 7efe667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.305+0000 7efe667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.305+0000 7efe667fc640 10 monclient: dump: 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 5 monclient: authenticate success, global_id 4305 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 10 monclient: _renew_subs 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7013a790 con 0x7efe701455a0 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6803e450 con 0x7efe701455a0 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7f2b99bdb640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f2b7806c820 0x7f2b7806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7f2b99bdb640 10 monclient: get_auth_request con 0x7f2b7806c820 auth_method 0 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe67fff640 4 set_mon_vals no callback set 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6803e7e0 con 0x7efe701455a0 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe667fc640 10 monclient: dump: 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.398 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 10 monclient: _renew_subs 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe701d7c70 con 0x7efe701455a0 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7efe77b7c640 1 librados: init done 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7f2b8a7fc640 10 client.4299.objecter ms_handle_connect 0x7f2b7806c820 2023-12-07T23:46:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.306+0000 7f2b99bdb640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f2b7806c820 0x7f2b7806ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006a90 tx=0x7f2b90006a20 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.307+0000 7efe77b7c640 10 client.4305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.307+0000 7efe77b7c640 15 client.4305.objecter _session_command_op_assign -1 1 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.307+0000 7efe77b7c640 20 client.4305.objecter _get_session osd=-1 returning homeless 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.307+0000 7efe77b7c640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.307+0000 7efe77b7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.307+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efe70142fe0 con 0x7efe701455a0 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.308+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== osd.3 v2:172.21.15.129:6808/2348435352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcf48075720 con 0x7fcf48072c50 2023-12-07T23:46:16.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.308+0000 7fcf437fe640 10 client.4250.objecter ms_dispatch 0x7fcf5c000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.308+0000 7fcf437fe640 10 client.4250.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.308+0000 7fcf437fe640 20 client.4250.objecter put_session s=0x7fcf48072580 osd=3 4 2023-12-07T23:46:16.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.308+0000 7fcf437fe640 15 client.4250.objecter _session_command_op_remove 3 1 2023-12-07T23:46:16.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2b84058f40 con 0x7f2b940a9bd0 2023-12-07T23:46:16.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 10 client.4299.objecter ms_dispatch 0x7f2b94000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 3 client.4299.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 3 client.4299.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 10 client.4299.objecter checking command 1 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f2b78072c00 0x7f2b78075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 20 client.4299.objecter _get_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 20 client.4299.objecter put_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 15 client.4299.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 20 client.4299.objecter _get_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 20 client.4299.objecter get_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 15 client.4299.objecter _session_command_op_assign 4 1 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 20 client.4299.objecter put_session s=0x7f2b78072530 osd=4 4 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 10 client.4299.objecter _send_command 1 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 --> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b780756d0 con 0x7f2b78072c00 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 20 client.4299.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b84054b80 con 0x7f2b940a9bd0 2023-12-07T23:46:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b8a7fc640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.312+0000 7f2b9a3dc640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f2b78072c00 0x7f2b78075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.313+0000 7f2b9a3dc640 10 monclient: get_auth_request con 0x7f2b78072c00 auth_method 0 2023-12-07T23:46:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.316+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7b64013470 con 0x7f7b680710e0 2023-12-07T23:46:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.316+0000 7f7b5effd640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b4c06c870 0x7f7b4c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.316+0000 7f2b9a3dc640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f2b78072c00 0x7f2b78075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.316+0000 7f2b8a7fc640 10 client.4299.objecter ms_handle_connect 0x7f2b78072c00 2023-12-07T23:46:16.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.316+0000 7f7b6dc17640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b4c06c870 0x7f7b4c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.316+0000 7f7b6dc17640 10 monclient: get_auth_request con 0x7f7b4c06c870 auth_method 0 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.317+0000 7f7b5effd640 10 client.4256.objecter ms_handle_connect 0x7f7b4c06c870 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.317+0000 7f7b6dc17640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b4c06c870 0x7f7b4c06ed30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7b681d7020 tx=0x7f7b58007990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7b64059000 con 0x7f7b680710e0 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 10 client.4256.objecter ms_dispatch 0x7f7b68000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 3 client.4256.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 3 client.4256.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 10 client.4256.objecter checking command 1 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7f7b4c072c50 0x7f7b4c075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 20 client.4256.objecter _get_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 20 client.4256.objecter put_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 15 client.4256.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 20 client.4256.objecter _get_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 20 client.4256.objecter get_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 15 client.4256.objecter _session_command_op_assign 2 1 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 20 client.4256.objecter put_session s=0x7f7b4c072580 osd=2 4 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 10 client.4256.objecter _send_command 1 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 --> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b4c075720 con 0x7f7b4c072c50 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 20 client.4256.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b64054c40 con 0x7f7b680710e0 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.321+0000 7f7b5effd640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.322+0000 7f7b6ec19640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7f7b4c072c50 0x7f7b4c075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.323+0000 7f7b6ec19640 10 monclient: get_auth_request con 0x7f7b4c072c50 auth_method 0 2023-12-07T23:46:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 10 client.4250.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 15 client.4250.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 20 client.4250.objecter _get_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 20 client.4250.objecter put_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 20 client.4250.objecter _get_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 15 client.4250.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 20 client.4250.objecter get_session s=0x7fcf48072580 osd=3 3 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 15 client.4250.objecter _session_command_op_assign 3 2 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 20 client.4250.objecter put_session s=0x7fcf48072580 osd=3 4 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 10 client.4250.objecter _send_command 2 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.325+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 --> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf540cba50 con 0x7fcf48072c50 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.327+0000 7f7b6ec19640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7f7b4c072c50 0x7f7b4c075070 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.327+0000 7f7b5effd640 10 client.4256.objecter ms_handle_connect 0x7f7b4c072c50 2023-12-07T23:46:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf437fe640 1 -- 172.21.15.129:0/1536799848 <== osd.3 v2:172.21.15.129:6808/2348435352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcf540cba50 con 0x7fcf48072c50 2023-12-07T23:46:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf437fe640 10 client.4250.objecter ms_dispatch 0x7fcf5c000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf437fe640 10 client.4250.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf437fe640 20 client.4250.objecter put_session s=0x7fcf48072580 osd=3 4 2023-12-07T23:46:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf437fe640 15 client.4250.objecter _session_command_op_remove 3 2 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf5bfff640 10 librados: watch_flush enter 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf5bfff640 10 librados: watch_flush exit 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.335+0000 7fcf5bfff640 10 client.4250.objecter close_session for osd.3 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7fcf48072c50 msgr2=0x7fcf48075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7fcf48072c50 0x7fcf48075070 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 20 client.4250.objecter put_session s=0x7fcf48072580 osd=3 1 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 20 client.4250.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 10 client.4250.objecter successfully canceled tick 2023-12-07T23:46:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== osd.0 v2:172.21.15.129:6824/771847878 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe400075750 con 0x7fe400072c80 2023-12-07T23:46:16.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fe411ffb640 10 client.4312.objecter ms_dispatch 0x7fe41c000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fe411ffb640 10 client.4312.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fe411ffb640 20 client.4312.objecter put_session s=0x7fe4000725b0 osd=0 4 2023-12-07T23:46:16.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fe411ffb640 15 client.4312.objecter _session_command_op_remove 0 1 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fcf4806c870 msgr2=0x7fcf4806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fcf4806c870 0x7fcf4806ed30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cf290 tx=0x7fcf4c009500 comp rx=0 tx=0).stop 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 10 monclient: shutdown 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 msgr2=0x7fcf540ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 --2- 172.21.15.129:0/1536799848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf54006c10 0x7fcf540ca860 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009890 tx=0x7fcf500046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5b7fe640 1 -- 172.21.15.129:0/1536799848 reap_dead start 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 shutdown_connections 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 >> 172.21.15.129:0/1536799848 conn(0x7fcf54005e80 msgr2=0x7fcf54004980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 shutdown_connections 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 -- 172.21.15.129:0/1536799848 wait complete. 2023-12-07T23:46:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.336+0000 7fcf5bfff640 1 librados: shutdown 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.340+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== osd.5 v2:172.21.15.184:6818/4213359653 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7efcf0075720 con 0x7efcf0072c50 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.340+0000 7efd00ff9640 10 client.4315.objecter ms_dispatch 0x7efd04000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.340+0000 7efd00ff9640 10 client.4315.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.340+0000 7efd00ff9640 20 client.4315.objecter put_session s=0x7efcf0072580 osd=5 4 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.340+0000 7efd00ff9640 15 client.4315.objecter _session_command_op_remove 5 1 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.346+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== osd.1 v2:172.21.15.129:6816/3964479910 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff798075720 con 0x7ff798072c50 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.346+0000 7ff7937fe640 10 client.4253.objecter ms_dispatch 0x7ff7ac000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.346+0000 7ff7937fe640 10 client.4253.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.346+0000 7ff7937fe640 20 client.4253.objecter put_session s=0x7ff798072580 osd=1 4 2023-12-07T23:46:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.346+0000 7ff7937fe640 15 client.4253.objecter _session_command_op_remove 1 1 2023-12-07T23:46:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.351+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe74c072320 con 0x7fe7440c53e0 2023-12-07T23:46:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.351+0000 7fe7337fe640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe73806c820 0x7fe73806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.414 INFO:teuthology.orchestra.run.smithi129.stdout:68719476742 2023-12-07T23:46:16.415 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.351+0000 7fe74affd640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe73806c820 0x7fe73806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.415 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.351+0000 7fe74affd640 10 monclient: get_auth_request con 0x7fe73806c820 auth_method 0 2023-12-07T23:46:16.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-07T23:46:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.352+0000 7fe7337fe640 10 client.4302.objecter ms_handle_connect 0x7fe73806c820 2023-12-07T23:46:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.352+0000 7fe74affd640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe73806c820 0x7fe73806ece0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe73c005020 tx=0x7fe73c007ef0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 10 client.4312.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 15 client.4312.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 20 client.4312.objecter _get_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 20 client.4312.objecter put_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 20 client.4312.objecter _get_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 15 client.4312.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 20 client.4312.objecter get_session s=0x7fe4000725b0 osd=0 3 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 15 client.4312.objecter _session_command_op_assign 0 2 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 20 client.4312.objecter put_session s=0x7fe4000725b0 osd=0 4 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 10 client.4312.objecter _send_command 2 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.356+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 --> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe4140370a0 con 0x7fe400072c80 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe411ffb640 1 -- 172.21.15.129:0/175273291 <== osd.0 v2:172.21.15.129:6824/771847878 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe418031a90 con 0x7fe400072c80 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe411ffb640 10 client.4312.objecter ms_dispatch 0x7fe41c000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe411ffb640 10 client.4312.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe411ffb640 20 client.4312.objecter put_session s=0x7fe4000725b0 osd=0 4 2023-12-07T23:46:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe411ffb640 15 client.4312.objecter _session_command_op_remove 0 2 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 10 librados: watch_flush enter 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 10 librados: watch_flush exit 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 10 client.4312.objecter close_session for osd.0 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7fe400072c80 msgr2=0x7fe4000750a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7fe400072c80 0x7fe4000750a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 20 client.4312.objecter put_session s=0x7fe4000725b0 osd=0 1 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 20 client.4312.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 10 client.4312.objecter successfully canceled tick 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe40006c8a0 msgr2=0x7fe40006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe40006c8a0 0x7fe40006ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe40c007fd0 tx=0x7fe40c009040 comp rx=0 tx=0).stop 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 10 monclient: shutdown 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe414006fa0 msgr2=0x7fe4140c8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.363+0000 7fe4225fb640 1 --2- 172.21.15.129:0/175273291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe414006fa0 0x7fe4140c8d40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe41c064260 tx=0x7fe41c0678c0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7fe421dfa640 1 -- 172.21.15.129:0/175273291 reap_dead start 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 shutdown_connections 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== osd.4 v2:172.21.15.184:6826/917696104 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2b780756d0 con 0x7f2b78072c00 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7f2b8a7fc640 10 client.4299.objecter ms_dispatch 0x7f2b94000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7f2b8a7fc640 10 client.4299.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7f2b8a7fc640 20 client.4299.objecter put_session s=0x7f2b78072530 osd=4 4 2023-12-07T23:46:16.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7f2b8a7fc640 15 client.4299.objecter _session_command_op_remove 4 1 2023-12-07T23:46:16.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 >> 172.21.15.129:0/175273291 conn(0x7fe414006170 msgr2=0x7fe4140cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 shutdown_connections 2023-12-07T23:46:16.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.364+0000 7fe4225fb640 1 -- 172.21.15.129:0/175273291 wait complete. 2023-12-07T23:46:16.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.365+0000 7fe4225fb640 1 librados: shutdown 2023-12-07T23:46:16.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.369+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efe6802cd50 con 0x7efe701455a0 2023-12-07T23:46:16.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.370+0000 7efe667fc640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efe5406c820 0x7efe5406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.370+0000 7efe75918640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efe5406c820 0x7efe5406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.370+0000 7efe75918640 10 monclient: get_auth_request con 0x7efe5406c820 auth_method 0 2023-12-07T23:46:16.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.370+0000 7efe75918640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efe5406c820 0x7efe5406ece0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe6c006a90 tx=0x7efe6c006a20 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.370+0000 7efe667fc640 10 client.4305.objecter ms_handle_connect 0x7efe5406c820 2023-12-07T23:46:16.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 10 client.4315.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 15 client.4315.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 20 client.4315.objecter _get_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 20 client.4315.objecter put_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 20 client.4315.objecter _get_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 15 client.4315.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 20 client.4315.objecter get_session s=0x7efcf0072580 osd=5 3 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 15 client.4315.objecter _session_command_op_assign 5 2 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 20 client.4315.objecter put_session s=0x7efcf0072580 osd=5 4 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 10 client.4315.objecter _send_command 2 2023-12-07T23:46:16.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 --> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd04144330 con 0x7efcf0072c50 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe74c0d2340 con 0x7fe7440c53e0 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 10 client.4302.objecter ms_dispatch 0x7fe74c000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 3 client.4302.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 3 client.4302.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 10 client.4302.objecter checking command 1 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fe738072c00 0x7fe738075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 20 client.4302.objecter _get_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 20 client.4302.objecter put_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 15 client.4302.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 20 client.4302.objecter _get_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 20 client.4302.objecter get_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 15 client.4302.objecter _session_command_op_assign 6 1 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 20 client.4302.objecter put_session s=0x7fe738072530 osd=6 4 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 10 client.4302.objecter _send_command 1 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 --> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7380756d0 con 0x7fe738072c00 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 20 client.4302.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe74c0ce680 con 0x7fe7440c53e0 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.374+0000 7fe7337fe640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.375+0000 7fe74a7fc640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fe738072c00 0x7fe738075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.375+0000 7fe74a7fc640 10 monclient: get_auth_request con 0x7fe738072c00 auth_method 0 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 10 client.4253.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 15 client.4253.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 20 client.4253.objecter _get_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 20 client.4253.objecter put_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 20 client.4253.objecter _get_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 15 client.4253.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 20 client.4253.objecter get_session s=0x7ff798072580 osd=1 3 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 15 client.4253.objecter _session_command_op_assign 1 2 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 20 client.4253.objecter put_session s=0x7ff798072580 osd=1 4 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 10 client.4253.objecter _send_command 2 2023-12-07T23:46:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.376+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7a40c5740 con 0x7ff798072c50 2023-12-07T23:46:16.427 INFO:teuthology.orchestra.run.smithi129.stdout:68719476742 2023-12-07T23:46:16.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-07T23:46:16.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.378+0000 7fe7337fe640 10 client.4302.objecter ms_handle_connect 0x7fe738072c00 2023-12-07T23:46:16.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.378+0000 7fe74a7fc640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fe738072c00 0x7fe738075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.380+0000 7efd00ff9640 1 -- 172.21.15.129:0/1232236853 <== osd.5 v2:172.21.15.184:6818/4213359653 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efd04144330 con 0x7efcf0072c50 2023-12-07T23:46:16.429 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.380+0000 7efd00ff9640 10 client.4315.objecter ms_dispatch 0x7efd04000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.429 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.380+0000 7efd00ff9640 10 client.4315.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.429 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.380+0000 7efd00ff9640 20 client.4315.objecter put_session s=0x7efcf0072580 osd=5 4 2023-12-07T23:46:16.429 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.380+0000 7efd00ff9640 15 client.4315.objecter _session_command_op_remove 5 2 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 10 librados: watch_flush enter 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 10 librados: watch_flush exit 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 10 client.4315.objecter close_session for osd.5 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7efcf0072c50 msgr2=0x7efcf0075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7efcf0072c50 0x7efcf0075070 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 20 client.4315.objecter put_session s=0x7efcf0072580 osd=5 1 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 20 client.4315.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 10 client.4315.objecter successfully canceled tick 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efcf006c870 msgr2=0x7efcf006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efcf006c870 0x7efcf006ed30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efcfc035000 tx=0x7efcfc030000 comp rx=0 tx=0).stop 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 10 monclient: shutdown 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd040aee70 msgr2=0x7efd040b0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 --2- 172.21.15.129:0/1232236853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd040aee70 0x7efd040b0de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c460 tx=0x7efcf800c920 comp rx=0 tx=0).stop 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd08e89640 1 -- 172.21.15.129:0/1232236853 reap_dead start 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 shutdown_connections 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 >> 172.21.15.129:0/1232236853 conn(0x7efd040b0630 msgr2=0x7efd0413b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 shutdown_connections 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.381+0000 7efd0968a640 1 -- 172.21.15.129:0/1232236853 wait complete. 2023-12-07T23:46:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7efd0968a640 1 librados: shutdown 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 10 client.4299.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 15 client.4299.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 20 client.4299.objecter _get_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 20 client.4299.objecter put_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 20 client.4299.objecter _get_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 15 client.4299.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 20 client.4299.objecter get_session s=0x7f2b78072530 osd=4 3 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 15 client.4299.objecter _session_command_op_assign 4 2 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 20 client.4299.objecter put_session s=0x7f2b78072530 osd=4 4 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 10 client.4299.objecter _send_command 2 2023-12-07T23:46:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.382+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 --> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2b941453f0 con 0x7f2b78072c00 2023-12-07T23:46:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efe68050e70 con 0x7efe701455a0 2023-12-07T23:46:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 10 client.4305.objecter ms_dispatch 0x7efe70000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 3 client.4305.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 3 client.4305.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 10 client.4305.objecter checking command 1 2023-12-07T23:46:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7ff7937fe640 1 -- 172.21.15.129:0/1037897232 <== osd.1 v2:172.21.15.129:6816/3964479910 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff7a0009870 con 0x7ff798072c50 2023-12-07T23:46:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7ff7937fe640 10 client.4253.objecter ms_dispatch 0x7ff7ac000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efe54072c00 0x7efe54075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7ff7937fe640 10 client.4253.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7ff7937fe640 20 client.4253.objecter put_session s=0x7ff798072580 osd=1 4 2023-12-07T23:46:16.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7ff7937fe640 15 client.4253.objecter _session_command_op_remove 1 2 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 20 client.4305.objecter _get_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 20 client.4305.objecter put_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 15 client.4305.objecter _session_command_op_remove -1 1 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 20 client.4305.objecter _get_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 20 client.4305.objecter get_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 15 client.4305.objecter _session_command_op_assign 7 1 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 20 client.4305.objecter put_session s=0x7efe54072530 osd=7 4 2023-12-07T23:46:16.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 10 client.4305.objecter _send_command 1 2023-12-07T23:46:16.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 --> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe540756d0 con 0x7efe54072c00 2023-12-07T23:46:16.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 20 client.4305.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe68077680 con 0x7efe701455a0 2023-12-07T23:46:16.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.384+0000 7efe667fc640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 10 librados: watch_flush enter 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 10 librados: watch_flush exit 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 10 client.4253.objecter close_session for osd.1 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7ff798072c50 msgr2=0x7ff798075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7ff798072c50 0x7ff798075070 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 20 client.4253.objecter put_session s=0x7ff798072580 osd=1 1 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 20 client.4253.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 10 client.4253.objecter successfully canceled tick 2023-12-07T23:46:16.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7efe75117640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efe54072c00 0x7efe54075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ff79806c870 msgr2=0x7ff79806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ff79806c870 0x7ff79806ed30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac02cee0 tx=0x7ff7ac068400 comp rx=0 tx=0).stop 2023-12-07T23:46:16.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 10 monclient: shutdown 2023-12-07T23:46:16.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a4033d30 msgr2=0x7ff7a40ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 --2- 172.21.15.129:0/1037897232 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff7a4033d30 0x7ff7a40ca030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff79c000c00 tx=0x7ff79c0046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7abfff640 1 -- 172.21.15.129:0/1037897232 reap_dead start 2023-12-07T23:46:16.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 shutdown_connections 2023-12-07T23:46:16.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 >> 172.21.15.129:0/1037897232 conn(0x7ff7a4006550 msgr2=0x7ff7a4032130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7efe75117640 10 monclient: get_auth_request con 0x7efe54072c00 auth_method 0 2023-12-07T23:46:16.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 shutdown_connections 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 -- 172.21.15.129:0/1037897232 wait complete. 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.385+0000 7ff7b16b4640 1 librados: shutdown 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.387+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== osd.2 v2:172.21.15.129:6800/262618579 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7b4c075720 con 0x7f7b4c072c50 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.387+0000 7f7b5effd640 10 client.4256.objecter ms_dispatch 0x7f7b68000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.387+0000 7f7b5effd640 10 client.4256.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.387+0000 7f7b5effd640 20 client.4256.objecter put_session s=0x7f7b4c072580 osd=2 4 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.387+0000 7f7b5effd640 15 client.4256.objecter _session_command_op_remove 2 1 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.389+0000 7efe667fc640 10 client.4305.objecter ms_handle_connect 0x7efe54072c00 2023-12-07T23:46:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.389+0000 7efe75117640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efe54072c00 0x7efe54075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.390+0000 7f2b8a7fc640 1 -- 172.21.15.129:0/2010684625 <== osd.4 v2:172.21.15.184:6826/917696104 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2b8c02c940 con 0x7f2b78072c00 2023-12-07T23:46:16.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.390+0000 7f2b8a7fc640 10 client.4299.objecter ms_dispatch 0x7f2b94000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.390+0000 7f2b8a7fc640 10 client.4299.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.390+0000 7f2b8a7fc640 20 client.4299.objecter put_session s=0x7f2b78072530 osd=4 4 2023-12-07T23:46:16.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.390+0000 7f2b8a7fc640 15 client.4299.objecter _session_command_op_remove 4 2 2023-12-07T23:46:16.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 10 librados: watch_flush enter 2023-12-07T23:46:16.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 10 librados: watch_flush exit 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 10 client.4299.objecter close_session for osd.4 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f2b78072c00 msgr2=0x7f2b78075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f2b78072c00 0x7f2b78075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 20 client.4299.objecter put_session s=0x7f2b78072530 osd=4 1 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 20 client.4299.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 10 client.4299.objecter successfully canceled tick 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f2b7806c820 msgr2=0x7f2b7806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f2b7806c820 0x7f2b7806ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006a90 tx=0x7f2b90006a20 comp rx=0 tx=0).stop 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 10 monclient: shutdown 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b940a9bd0 msgr2=0x7f2b940b52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 --2- 172.21.15.129:0/2010684625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b940a9bd0 0x7f2b940b52c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e980 tx=0x7f2b8400ee40 comp rx=0 tx=0).stop 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9a3dc640 1 -- 172.21.15.129:0/2010684625 reap_dead start 2023-12-07T23:46:16.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 shutdown_connections 2023-12-07T23:46:16.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 >> 172.21.15.129:0/2010684625 conn(0x7f2b94066dd0 msgr2=0x7f2b940b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.391+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 shutdown_connections 2023-12-07T23:46:16.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.392+0000 7f2b9be3f640 1 -- 172.21.15.129:0/2010684625 wait complete. 2023-12-07T23:46:16.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.392+0000 7f2b9be3f640 1 librados: shutdown 2023-12-07T23:46:16.444 INFO:teuthology.orchestra.run.smithi129.stdout:55834574853 2023-12-07T23:46:16.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-07T23:46:16.445 INFO:teuthology.orchestra.run.smithi129.stdout:60129542148 2023-12-07T23:46:16.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-07T23:46:16.447 INFO:teuthology.orchestra.run.smithi129.stdout:55834574852 2023-12-07T23:46:16.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 15 client.4256.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 20 client.4256.objecter _get_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 20 client.4256.objecter put_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 20 client.4256.objecter _get_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 15 client.4256.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 20 client.4256.objecter get_session s=0x7f7b4c072580 osd=2 3 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 15 client.4256.objecter _session_command_op_assign 2 2 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 20 client.4256.objecter put_session s=0x7f7b4c072580 osd=2 4 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 10 client.4256.objecter _send_command 2 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.409+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 --> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7b68142730 con 0x7f7b4c072c50 2023-12-07T23:46:16.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b5effd640 1 -- 172.21.15.129:0/1597030079 <== osd.2 v2:172.21.15.129:6800/262618579 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7b68142730 con 0x7f7b4c072c50 2023-12-07T23:46:16.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b5effd640 10 client.4256.objecter ms_dispatch 0x7f7b68000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b5effd640 10 client.4256.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b5effd640 20 client.4256.objecter put_session s=0x7f7b4c072580 osd=2 4 2023-12-07T23:46:16.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b5effd640 15 client.4256.objecter _session_command_op_remove 2 2 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 10 librados: watch_flush enter 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 10 librados: watch_flush exit 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 10 client.4256.objecter close_session for osd.2 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7f7b4c072c50 msgr2=0x7f7b4c075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7f7b4c072c50 0x7f7b4c075070 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 20 client.4256.objecter put_session s=0x7f7b4c072580 osd=2 1 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 20 client.4256.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 10 client.4256.objecter successfully canceled tick 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b4c06c870 msgr2=0x7f7b4c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b4c06c870 0x7f7b4c06ed30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7b681d7020 tx=0x7f7b58007990 comp rx=0 tx=0).stop 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 10 monclient: shutdown 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680710e0 msgr2=0x7f7b680b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 --2- 172.21.15.129:0/1597030079 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b680710e0 0x7f7b680b3670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400e9b0 tx=0x7f7b6400ee70 comp rx=0 tx=0).stop 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b6ec19640 1 -- 172.21.15.129:0/1597030079 reap_dead start 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 shutdown_connections 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 >> 172.21.15.129:0/1597030079 conn(0x7f7b680b4c20 msgr2=0x7f7b680b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 shutdown_connections 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 -- 172.21.15.129:0/1597030079 wait complete. 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.414+0000 7f7b7067c640 1 librados: shutdown 2023-12-07T23:46:16.452 INFO:teuthology.orchestra.run.smithi129.stdout:68719476742 2023-12-07T23:46:16.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-07T23:46:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.429+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== osd.6 v2:172.21.15.184:6807/105483051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe7380756d0 con 0x7fe738072c00 2023-12-07T23:46:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.429+0000 7fe7337fe640 10 client.4302.objecter ms_dispatch 0x7fe74c000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.429+0000 7fe7337fe640 10 client.4302.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.429+0000 7fe7337fe640 20 client.4302.objecter put_session s=0x7fe738072530 osd=6 4 2023-12-07T23:46:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.429+0000 7fe7337fe640 15 client.4302.objecter _session_command_op_remove 6 1 2023-12-07T23:46:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.439+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== osd.7 v2:172.21.15.184:6802/59760990 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7efe540756d0 con 0x7efe54072c00 2023-12-07T23:46:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.439+0000 7efe667fc640 10 client.4305.objecter ms_dispatch 0x7efe70000b90 command_reply(tid 1: 0 ) v1 2023-12-07T23:46:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.439+0000 7efe667fc640 10 client.4305.objecter _finish_command 1 = system:0 2023-12-07T23:46:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.439+0000 7efe667fc640 20 client.4305.objecter put_session s=0x7efe54072530 osd=7 4 2023-12-07T23:46:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.439+0000 7efe667fc640 15 client.4305.objecter _session_command_op_remove 7 1 2023-12-07T23:46:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 10 client.4302.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 15 client.4302.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 20 client.4302.objecter _get_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 20 client.4302.objecter put_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 20 client.4302.objecter _get_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 15 client.4302.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 20 client.4302.objecter get_session s=0x7fe738072530 osd=6 3 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 15 client.4302.objecter _session_command_op_assign 6 2 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 20 client.4302.objecter put_session s=0x7fe738072530 osd=6 4 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 10 client.4302.objecter _send_command 2 2023-12-07T23:46:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.443+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 --> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe7440c5830 con 0x7fe738072c00 2023-12-07T23:46:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 10 client.4305.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T23:46:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 15 client.4305.objecter _session_command_op_assign -1 2 2023-12-07T23:46:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 20 client.4305.objecter _get_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 20 client.4305.objecter put_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 20 client.4305.objecter _get_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 15 client.4305.objecter _session_command_op_remove -1 2 2023-12-07T23:46:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 20 client.4305.objecter get_session s=0x7efe54072530 osd=7 3 2023-12-07T23:46:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 15 client.4305.objecter _session_command_op_assign 7 2 2023-12-07T23:46:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 20 client.4305.objecter put_session s=0x7efe54072530 osd=7 4 2023-12-07T23:46:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 10 client.4305.objecter _send_command 2 2023-12-07T23:46:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 --> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efe701d6d30 con 0x7efe54072c00 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7337fe640 1 -- 172.21.15.129:0/2131958372 <== osd.6 v2:172.21.15.184:6807/105483051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe740031bc0 con 0x7fe738072c00 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7337fe640 10 client.4302.objecter ms_dispatch 0x7fe74c000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7337fe640 10 client.4302.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7337fe640 20 client.4302.objecter put_session s=0x7fe738072530 osd=6 4 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7337fe640 15 client.4302.objecter _session_command_op_remove 6 2 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 10 librados: watch_flush enter 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 10 librados: watch_flush exit 2023-12-07T23:46:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 10 client.4302.objecter close_session for osd.6 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fe738072c00 msgr2=0x7fe738075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fe738072c00 0x7fe738075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 20 client.4302.objecter put_session s=0x7fe738072530 osd=6 1 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 20 client.4302.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 10 client.4302.objecter successfully canceled tick 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe73806c820 msgr2=0x7fe73806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe73806c820 0x7fe73806ece0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe73c005020 tx=0x7fe73c007ef0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 10 monclient: shutdown 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7440c53e0 msgr2=0x7fe744160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe7515a9640 1 --2- 172.21.15.129:0/2131958372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7440c53e0 0x7fe744160ad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe74c068930 tx=0x7fe74c0678c0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.454+0000 7fe74b7fe640 1 -- 172.21.15.129:0/2131958372 reap_dead start 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.455+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 shutdown_connections 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.455+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 >> 172.21.15.129:0/2131958372 conn(0x7fe744006a20 msgr2=0x7fe744004f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.455+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 shutdown_connections 2023-12-07T23:46:16.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.455+0000 7fe7515a9640 1 -- 172.21.15.129:0/2131958372 wait complete. 2023-12-07T23:46:16.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.455+0000 7fe7515a9640 1 librados: shutdown 2023-12-07T23:46:16.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.462+0000 7efe667fc640 1 -- 172.21.15.129:0/1903207237 <== osd.7 v2:172.21.15.184:6802/59760990 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efe701d6d30 con 0x7efe54072c00 2023-12-07T23:46:16.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.462+0000 7efe667fc640 10 client.4305.objecter ms_dispatch 0x7efe70000b90 command_reply(tid 2: 0 ) v1 2023-12-07T23:46:16.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.462+0000 7efe667fc640 10 client.4305.objecter _finish_command 2 = system:0 2023-12-07T23:46:16.465 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.462+0000 7efe667fc640 20 client.4305.objecter put_session s=0x7efe54072530 osd=7 4 2023-12-07T23:46:16.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.462+0000 7efe667fc640 15 client.4305.objecter _session_command_op_remove 7 2 2023-12-07T23:46:16.466 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.462+0000 7efe77b7c640 10 librados: watch_flush enter 2023-12-07T23:46:16.467 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 10 librados: watch_flush exit 2023-12-07T23:46:16.467 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 10 client.4305.objecter close_session for osd.7 2023-12-07T23:46:16.467 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efe54072c00 msgr2=0x7efe54075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.467 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efe54072c00 0x7efe54075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 20 client.4305.objecter put_session s=0x7efe54072530 osd=7 1 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 20 client.4305.objecter shutdown clearing up homeless session... 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 10 client.4305.objecter successfully canceled tick 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efe5406c820 msgr2=0x7efe5406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efe5406c820 0x7efe5406ece0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe6c006a90 tx=0x7efe6c006a20 comp rx=0 tx=0).stop 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 10 monclient: shutdown 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 msgr2=0x7efe701d7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.464+0000 7efe77b7c640 1 --2- 172.21.15.129:0/1903207237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe701455a0 0x7efe701d7550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe68036040 tx=0x7efe68004a40 comp rx=0 tx=0).stop 2023-12-07T23:46:16.468 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.465+0000 7efe76119640 1 -- 172.21.15.129:0/1903207237 reap_dead start 2023-12-07T23:46:16.469 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.465+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 shutdown_connections 2023-12-07T23:46:16.469 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.465+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 >> 172.21.15.129:0/1903207237 conn(0x7efe700b3ea0 msgr2=0x7efe700ad140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.469 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.466+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 shutdown_connections 2023-12-07T23:46:16.469 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.466+0000 7efe77b7c640 1 -- 172.21.15.129:0/1903207237 wait complete. 2023-12-07T23:46:16.469 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.466+0000 7efe77b7c640 1 librados: shutdown 2023-12-07T23:46:16.477 INFO:teuthology.orchestra.run.smithi129.stdout:55834574852 2023-12-07T23:46:16.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-07T23:46:16.494 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2023-12-07T23:46:16.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-07T23:46:16.619 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.607+0000 7f4a5eaec640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.607+0000 7f4a5eaec640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.607+0000 7f4a5eaec640 10 monclient: monmap: 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.608818+0000 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.608818+0000 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.620 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.608+0000 7f4a5eaec640 1 Processor -- start 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.608+0000 7f4a5eaec640 1 -- start start 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.608+0000 7f4a5eaec640 10 monclient: init 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a580b0980 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4a580abcb0 0x7f4a58152240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient(hunting): picked mon.noname-c con 0x7f4a580abcb0 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a58152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient(hunting): picked mon.noname-b con 0x7f4a58152780 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a580ac120 con 0x7f4a580b0980 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a581530f0 con 0x7f4a580abcb0 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580b0d60 con 0x7f4a58152780 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.609+0000 7f4a5eaec640 10 monclient(hunting): authenticate will time out at 1376.797871s 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.611+0000 7f4a57fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4a580abcb0 0x7f4a58152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.611+0000 7f4a5c888640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4a580b0980 0x7f4a580a9230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43120/0 (socket says 172.21.15.129:43120) 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.611+0000 7f4a5c888640 1 -- 172.21.15.129:0/3012981096 learned_addr learned my addr 172.21.15.129:0/3012981096 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.611+0000 7f4a5d089640 1 --2- 172.21.15.129:0/3012981096 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a58152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a577fe640 1 -- 172.21.15.129:0/3012981096 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419899429 0 0) 0x7f4a580ac120 con 0x7f4a580b0980 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.612+0000 7f4a5d089640 10 monclient(hunting): get_auth_request con 0x7f4a58152780 auth_method 0 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a5d089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a5d089640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a5d089640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a57fff640 10 monclient(hunting): get_auth_request con 0x7f4a580abcb0 auth_method 0 2023-12-07T23:46:16.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a57fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a57fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a577fe640 10 monclient(hunting): _negotiate 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a577fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a577fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a577fe640 10 monclient(hunting): my global_id is 4308 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.616+0000 7f4a577fe640 1 -- 172.21.15.129:0/3012981096 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a40003500 con 0x7f4a580b0980 2023-12-07T23:46:16.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.618+0000 7f4a57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.624 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.618+0000 7f4a57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.624 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.618+0000 7f4a57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.625 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.622+0000 7f4a5d089640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.625 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.622+0000 7f4a5d089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.625 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.622+0000 7f4a5d089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.622+0000 7f677678e640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.622+0000 7f677678e640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.622+0000 7f677678e640 10 monclient: monmap: 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.623508+0000 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.623508+0000 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.626 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.627 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.627 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.623+0000 7f677678e640 1 Processor -- start 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.623+0000 7f677678e640 1 -- start start 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.623+0000 7f677678e640 10 monclient: init 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f67700aee70 0x7f6770072100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): picked mon.noname-c con 0x7f67700aee70 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6770072640 0x7f6770141300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): picked mon.noname-a con 0x7f6770072640 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6770070210 0x7f6770141840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): picked mon.noname-b con 0x7f6770070210 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67700714c0 con 0x7f6770072640 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67700af250 con 0x7f67700aee70 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67700716c0 con 0x7f6770070210 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.624+0000 7f677678e640 10 monclient(hunting): authenticate will time out at 1376.812586s 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.626+0000 7f677578c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f67700aee70 0x7f6770072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.626+0000 7f677578c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f67700aee70 0x7f6770072100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40608/0 (socket says 172.21.15.129:40608) 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.626+0000 7f677578c640 1 -- 172.21.15.129:0/646283609 learned_addr learned my addr 172.21.15.129:0/646283609 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.626+0000 7f677578c640 10 monclient(hunting): get_auth_request con 0x7f67700aee70 auth_method 0 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.626+0000 7f677578c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.626+0000 7f677578c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.631 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.626+0000 7f677578c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 1 -- 172.21.15.129:0/3012981096 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 msgr2=0x7f4a58152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 1 --2- 172.21.15.129:0/3012981096 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a58152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 1 -- 172.21.15.129:0/3012981096 >> v1:172.21.15.129:6789/0 conn(0x7f4a580b0980 legacy=0x7f4a580a9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.625+0000 7f4a57fff640 1 -- 172.21.15.129:0/3012981096 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580a9940 con 0x7f4a580abcb0 2023-12-07T23:46:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.627+0000 7f9f6d232640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.628+0000 7f4a57fff640 1 --2- 172.21.15.129:0/3012981096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a580abcb0 0x7f4a58152240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c310 tx=0x7f4a500306d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39714fe196a239e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.634 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.627+0000 7f9f6d232640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.634 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.627+0000 7f9f6d232640 10 monclient: monmap: 2023-12-07T23:46:16.634 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.634 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.634 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.628406+0000 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.628406+0000 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.628+0000 7f9f6d232640 1 Processor -- start 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.628+0000 7f9f6d232640 1 -- start start 2023-12-07T23:46:16.635 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.628+0000 7f9f6d232640 10 monclient: init 2023-12-07T23:46:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6775f8d640 1 --2- 172.21.15.129:0/646283609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6770070210 0x7f6770141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6775f8d640 10 monclient(hunting): get_auth_request con 0x7f6770070210 auth_method 0 2023-12-07T23:46:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6775f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6775f8d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6775f8d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.637 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.637 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f68072430 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9f68072810 0x7f9f68141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient(hunting): picked mon.noname-c con 0x7f9f68072810 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9f680a8b80 0x7f9f6814c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient(hunting): picked mon.noname-a con 0x7f9f680a8b80 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f680aace0 con 0x7f9f68072430 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6774f8b640 1 --2- 172.21.15.129:0/646283609 >> v2:172.21.15.129:3300/0 conn(0x7f6770072640 0x7f6770141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.639 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6774f8b640 10 monclient(hunting): get_auth_request con 0x7f6770072640 auth_method 0 2023-12-07T23:46:16.639 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6774f8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.639 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6774f8b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.639 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f6774f8b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.639 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680aafc0 con 0x7f9f680a8b80 2023-12-07T23:46:16.640 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f677578c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.640 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f677578c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.640 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f677578c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680a9360 con 0x7f9f68072810 2023-12-07T23:46:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.629+0000 7f9f6d232640 10 monclient(hunting): authenticate will time out at 1376.817377s 2023-12-07T23:46:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.630+0000 7f6775f8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.630+0000 7f6775f8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.630+0000 7f6775f8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.630+0000 7f6774f8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.630+0000 7f6774f8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.630+0000 7f6774f8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.643 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.631+0000 7f9f6759e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9f680a8b80 0x7f9f6814c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.631+0000 7f379df16640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.631+0000 7f379df16640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.631+0000 7f379df16640 10 monclient: monmap: 2023-12-07T23:46:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.632762+0000 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.632762+0000 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.646 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 1 -- 172.21.15.129:0/3012981096 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5000a4e0 con 0x7f4a580abcb0 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 10 monclient: dump: 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a5eaec640 5 monclient: authenticate success, global_id 4324 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a5eaec640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 1 -- 172.21.15.129:0/3012981096 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50033050 con 0x7f4a580abcb0 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.632+0000 7f4a577fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 1 -- 172.21.15.129:0/646283609 >> v2:172.21.15.184:3300/0 conn(0x7f67700aee70 msgr2=0x7f6770072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 1 --2- 172.21.15.129:0/646283609 >> v2:172.21.15.184:3300/0 conn(0x7f67700aee70 0x7f6770072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 1 -- 172.21.15.129:0/646283609 >> v2:172.21.15.129:3300/0 conn(0x7f6770072640 msgr2=0x7f6770141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 1 --2- 172.21.15.129:0/646283609 >> v2:172.21.15.129:3300/0 conn(0x7f6770072640 0x7f6770141300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f4a5eaec640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f4a5eaec640 4 set_mon_vals no callback set 2023-12-07T23:46:16.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f9f6659c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9f68072810 0x7f9f68141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f9f6659c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9f68072810 0x7f9f68141ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40614/0 (socket says 172.21.15.129:40614) 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f9f6659c640 1 -- 172.21.15.129:0/4136254367 learned_addr learned my addr 172.21.15.129:0/4136254367 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f9f6659c640 10 monclient(hunting): get_auth_request con 0x7f9f68072810 auth_method 0 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f9f6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f9f6659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f9f6659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7f6775f8d640 1 -- 172.21.15.129:0/646283609 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6770071ae0 con 0x7f6770070210 2023-12-07T23:46:16.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 10 monclient: shutdown 2023-12-07T23:46:16.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3012981096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a580abcb0 msgr2=0x7f4a58152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3012981096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a580abcb0 0x7f4a58152240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c310 tx=0x7f4a500306d0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7ffb8ec2d640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7ffb8ec2d640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3012981096 shutdown_connections 2023-12-07T23:46:16.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3012981096 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a58152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3012981096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a580abcb0 0x7f4a58152240 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3012981096 >> 172.21.15.129:0/3012981096 conn(0x7f4a5806aaf0 msgr2=0x7f4a580b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.633+0000 7ffb8ec2d640 10 monclient: monmap: 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.635053+0000 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.635053+0000 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.654 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.655 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.655 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f379df16640 1 Processor -- start 2023-12-07T23:46:16.655 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f379df16640 1 -- start start 2023-12-07T23:46:16.655 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7f379df16640 10 monclient: init 2023-12-07T23:46:16.655 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.634+0000 7ffb8ec2d640 1 Processor -- start 2023-12-07T23:46:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3012981096 shutdown_connections 2023-12-07T23:46:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7ffb8ec2d640 1 -- start start 2023-12-07T23:46:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7ffb8ec2d640 10 monclient: init 2023-12-07T23:46:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-12-07T23:46:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37900071c0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f37900d01b0 0x7f3790030820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient(hunting): picked mon.noname-c con 0x7f37900d01b0 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3790033550 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37900375e0 con 0x7f37900071c0 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37900cf580 con 0x7f3790033550 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3790035120 con 0x7f37900d01b0 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f379df16640 10 monclient(hunting): authenticate will time out at 1376.823148s 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb880aef00 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ffb88143840 0x7ffb881391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb88143840 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb88139710 0x7ffb88139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb88139710 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb880a8990 con 0x7ffb880aef00 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb880af2e0 con 0x7ffb88139710 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb88071ae0 con 0x7ffb88143840 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7ffb8ec2d640 10 monclient(hunting): authenticate will time out at 1376.824301s 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.635+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3012981096 wait complete. 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 10 monclient: monmap: 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.659 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7fda84f00640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7fda84f00640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f9f6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 1 librados: starting msgr at 2023-12-07T23:46:16.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 1 librados: starting objecter 2023-12-07T23:46:16.661 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 1 Processor -- start 2023-12-07T23:46:16.661 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f9f6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.661 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f9f6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f9f6759e640 10 monclient(hunting): get_auth_request con 0x7f9f680a8b80 auth_method 0 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f9f6759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f9f6759e640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f9f6759e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7fda84f00640 10 monclient: monmap: 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.637836+0000 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.637836+0000 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.662 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.663 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.663 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.663 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.663 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.663 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.663 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.637+0000 7f6775f8d640 1 --2- 172.21.15.129:0/646283609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6770070210 0x7f6770141840 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6770072d30 tx=0x7f676002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8160e7bdfcc5d73 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 1 -- start start 2023-12-07T23:46:16.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 1 librados: setting wanted keys 2023-12-07T23:46:16.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 1 librados: calling monclient init 2023-12-07T23:46:16.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.636+0000 7f4a5eaec640 10 monclient: init 2023-12-07T23:46:16.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.637+0000 7fda84f00640 1 Processor -- start 2023-12-07T23:46:16.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.637+0000 7fda84f00640 1 -- start start 2023-12-07T23:46:16.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.637+0000 7fda84f00640 10 monclient: init 2023-12-07T23:46:16.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.637+0000 7f4a5eaec640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T23:46:16.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a580b0980 0x7f4a58079ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): picked mon.noname-a con 0x7f4a580b0980 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a5807ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): picked mon.noname-c con 0x7f4a58152780 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a5807b2d0 0x7f4a581e4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5807b2d0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a581e3c80 con 0x7f4a580b0980 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5806fb80 con 0x7f4a5807b2d0 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5806fd40 con 0x7f4a58152780 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f4a5eaec640 10 monclient(hunting): authenticate will time out at 1376.826450s 2023-12-07T23:46:16.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7ffb8d1ca640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb88139710 0x7ffb88139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-12-07T23:46:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda800b3390 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda80143be0 0x7fda80070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient(hunting): picked mon.noname-b con 0x7fda80143be0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fda8006e970 0x7fda80071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient(hunting): picked mon.noname-c con 0x7fda8006e970 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8006f100 con 0x7fda800b3390 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8006f450 con 0x7fda8006e970 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda800b3770 con 0x7fda80143be0 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7fda84f00640 10 monclient(hunting): authenticate will time out at 1376.826896s 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2023-12-07T23:46:16.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 1 -- 172.21.15.129:0/4136254367 >> v2:172.21.15.129:3300/0 conn(0x7f9f680a8b80 msgr2=0x7f9f6814c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 1 --2- 172.21.15.129:0/4136254367 >> v2:172.21.15.129:3300/0 conn(0x7f9f680a8b80 0x7f9f6814c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 1 -- 172.21.15.129:0/4136254367 >> v1:172.21.15.184:6789/0 conn(0x7f9f68072430 legacy=0x7f9f6813f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.638+0000 7f9f6659c640 1 -- 172.21.15.129:0/4136254367 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6813fa30 con 0x7f9f68072810 2023-12-07T23:46:16.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.640+0000 7ffb8d1ca640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb88139710 0x7ffb88139ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42894/0 (socket says 172.21.15.129:42894) 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.640+0000 7ffb8d1ca640 1 -- 172.21.15.129:0/4214633297 learned_addr learned my addr 172.21.15.129:0/4214633297 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.640+0000 7ffb8d1ca640 10 monclient(hunting): get_auth_request con 0x7ffb88139710 auth_method 0 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.640+0000 7ffb8d1ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.640+0000 7ffb8d1ca640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.640+0000 7ffb8d1ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.641+0000 7ffb8d1ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.641+0000 7ffb8d1ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.641+0000 7ffb8d1ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 1 -- 172.21.15.129:0/646283609 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600316f0 con 0x7f6770070210 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: dump: 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.673 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 1 -- 172.21.15.129:0/646283609 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6760031850 con 0x7f6770070210 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 1 -- 172.21.15.129:0/646283609 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6760031be0 con 0x7f6770070210 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f6767fff640 10 monclient: dump: 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f677678e640 5 monclient: authenticate success, global_id 4262 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f677678e640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7f677678e640 4 set_mon_vals no callback set 2023-12-07T23:46:16.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.642+0000 7fda77fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda80143be0 0x7fda80070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a5807ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f3796ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f37900d01b0 0x7f3790030820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 10 monclient: shutdown 2023-12-07T23:46:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 -- 172.21.15.129:0/646283609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6770070210 msgr2=0x7f6770141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 --2- 172.21.15.129:0/646283609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6770070210 0x7f6770141840 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6770072d30 tx=0x7f676002b790 comp rx=0 tx=0).stop 2023-12-07T23:46:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 -- 172.21.15.129:0/646283609 shutdown_connections 2023-12-07T23:46:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 --2- 172.21.15.129:0/646283609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6770070210 0x7f6770141840 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 --2- 172.21.15.129:0/646283609 >> v2:172.21.15.129:3300/0 conn(0x7f6770072640 0x7f6770141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 --2- 172.21.15.129:0/646283609 >> v2:172.21.15.184:3300/0 conn(0x7f67700aee70 0x7f6770072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 -- 172.21.15.129:0/646283609 >> 172.21.15.129:0/646283609 conn(0x7f67700b0630 msgr2=0x7f677006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f3796ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f37900d01b0 0x7f3790030820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40624/0 (socket says 172.21.15.129:40624) 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f3796ffd640 1 -- 172.21.15.129:0/688993352 learned_addr learned my addr 172.21.15.129:0/688993352 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f3796ffd640 10 monclient(hunting): get_auth_request con 0x7f37900d01b0 auth_method 0 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f3796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f3796ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f3796ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 -- 172.21.15.129:0/646283609 shutdown_connections 2023-12-07T23:46:16.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7fda77fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda80143be0 0x7fda80070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55592/0 (socket says 172.21.15.129:55592) 2023-12-07T23:46:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7fda77fff640 1 -- 172.21.15.129:0/1369892602 learned_addr learned my addr 172.21.15.129:0/1369892602 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7fda77fff640 10 monclient(hunting): get_auth_request con 0x7fda80143be0 auth_method 0 2023-12-07T23:46:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7fda77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7fda77fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7fda77fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7f677678e640 1 -- 172.21.15.129:0/646283609 wait complete. 2023-12-07T23:46:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2023-12-07T23:46:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 1 -- 172.21.15.129:0/4214633297 >> v2:172.21.15.184:3300/0 conn(0x7ffb88143840 msgr2=0x7ffb881391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 1 --2- 172.21.15.129:0/4214633297 >> v2:172.21.15.184:3300/0 conn(0x7ffb88143840 0x7ffb881391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 1 -- 172.21.15.129:0/4214633297 >> v1:172.21.15.184:6789/0 conn(0x7ffb880aef00 legacy=0x7ffb88143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.643+0000 7ffb8d1ca640 1 -- 172.21.15.129:0/4214633297 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb880a8c00 con 0x7ffb88139710 2023-12-07T23:46:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a5807ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55584/0 (socket says 172.21.15.129:55584) 2023-12-07T23:46:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f4a57fff640 1 -- 172.21.15.129:0/3075773185 learned_addr learned my addr 172.21.15.129:0/3075773185 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f4a57fff640 10 monclient(hunting): get_auth_request con 0x7f4a58152780 auth_method 0 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f4a57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f4a57fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f4a57fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f677678e640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f677678e640 10 monclient: monmap: 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f3796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f3796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f3796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f9f6659c640 1 --2- 172.21.15.129:0/4136254367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f68072810 0x7f9f68141ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9f680ad870 tx=0x7f9f5802b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e90af617f91d2c1e server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.686 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f677678e640 1 librados: starting msgr at 2023-12-07T23:46:16.686 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.644+0000 7f677678e640 1 librados: starting objecter 2023-12-07T23:46:16.686 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 1 Processor -- start 2023-12-07T23:46:16.686 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 1 -- start start 2023-12-07T23:46:16.686 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 1 librados: setting wanted keys 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 1 librados: calling monclient init 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 10 monclient: init 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6770070210 0x7f677013e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 10 monclient(hunting): picked mon.noname-b con 0x7f6770070210 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67700aee70 0x7f67701da5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 10 monclient(hunting): picked mon.noname-a con 0x7f67700aee70 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f677013d760 0x7f677013dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 10 monclient(hunting): picked mon.noname-c con 0x7f677013d760 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.645+0000 7f677678e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f677678e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677013c000 con 0x7f67700aee70 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f677678e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f677678e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67701daf70 con 0x7f6770070210 2023-12-07T23:46:16.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f677678e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f677678e640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677013e130 con 0x7f677013d760 2023-12-07T23:46:16.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f677678e640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f677678e640 10 monclient(hunting): authenticate will time out at 1376.834082s 2023-12-07T23:46:16.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f4a57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f4a57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7f4a57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7fda77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7fda77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.646+0000 7fda77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a5c888640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a580b0980 0x7f4a58079ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7ffb8d1ca640 1 --2- 172.21.15.129:0/4214633297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb88139ba0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb8806b330 tx=0x7ffb7802fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1883dd9290805ea5 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 1 -- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a5807b2d0 msgr2=0x7f4a581e4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a5807b2d0 0x7f4a581e4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 1 -- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a580b0980 msgr2=0x7f4a58079ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a580b0980 0x7f4a58079ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.649+0000 7f4a57fff640 1 -- 172.21.15.129:0/3075773185 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5000c050 con 0x7f4a58152780 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 1 -- 172.21.15.129:0/688993352 >> v1:172.21.15.184:6789/0 conn(0x7f3790033550 legacy=0x7f37900339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 1 -- 172.21.15.129:0/688993352 >> v1:172.21.15.129:6789/0 conn(0x7f37900071c0 legacy=0x7f37900cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7f3796ffd640 1 -- 172.21.15.129:0/688993352 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3790035260 con 0x7f37900d01b0 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.650+0000 7fda7ed9d640 1 --2- 172.21.15.129:0/1369892602 >> v2:172.21.15.184:3300/0 conn(0x7fda8006e970 0x7fda80071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.651+0000 7fda7ed9d640 10 monclient(hunting): get_auth_request con 0x7fda8006e970 auth_method 0 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.651+0000 7fda7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.651+0000 7fda7ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.651+0000 7fda7ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 1 -- 172.21.15.129:0/1369892602 >> v2:172.21.15.184:3300/0 conn(0x7fda8006e970 msgr2=0x7fda80071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 1 --2- 172.21.15.129:0/1369892602 >> v2:172.21.15.184:3300/0 conn(0x7fda8006e970 0x7fda80071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 1 -- 172.21.15.129:0/1369892602 >> v1:172.21.15.184:6789/0 conn(0x7fda800b3390 legacy=0x7fda801434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.694 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.694 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.694 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.652+0000 7fda77fff640 1 -- 172.21.15.129:0/1369892602 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800709a0 con 0x7fda80143be0 2023-12-07T23:46:16.694 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.654+0000 7f6775f8d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f677013d760 0x7f677013dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.694 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.656+0000 7fda77fff640 1 --2- 172.21.15.129:0/1369892602 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda80143be0 0x7fda80070dc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda680098a0 tx=0x7fda6802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f65a5bfb734a25d server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.695 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.656+0000 7f4a57fff640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a5807ad90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c430 tx=0x7f4a50004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.695 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.656+0000 7f4a557fa640 10 client.?.objecter ms_handle_connect 0x7f4a58152780 2023-12-07T23:46:16.695 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.656+0000 7f4a557fa640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.695 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 1 -- 172.21.15.129:0/4136254367 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58030870 con 0x7f9f68072810 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 10 monclient: dump: 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.696 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f6d232640 5 monclient: authenticate success, global_id 4330 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f6d232640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 1 -- 172.21.15.129:0/4136254367 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f580309d0 con 0x7f9f68072810 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f65d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f6d232640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.657+0000 7f9f6d232640 4 set_mon_vals no callback set 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f65d9b640 1 -- 172.21.15.129:0/4136254367 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58030d60 con 0x7f9f68072810 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f65d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f65d9b640 10 monclient: dump: 2023-12-07T23:46:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f6775f8d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f677013d760 0x7f677013dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55598/0 (socket says 172.21.15.129:55598) 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f6775f8d640 1 -- 172.21.15.129:0/732287447 learned_addr learned my addr 172.21.15.129:0/732287447 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.698 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f6775f8d640 10 monclient(hunting): get_auth_request con 0x7f677013d760 auth_method 0 2023-12-07T23:46:16.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f6775f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f6775f8d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f6775f8d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 10 monclient: shutdown 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 -- 172.21.15.129:0/4136254367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f68072810 msgr2=0x7f9f68141ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 --2- 172.21.15.129:0/4136254367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f68072810 0x7f9f68141ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9f680ad870 tx=0x7f9f5802b9b0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 -- 172.21.15.129:0/4136254367 shutdown_connections 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 --2- 172.21.15.129:0/4136254367 >> v2:172.21.15.129:3300/0 conn(0x7f9f680a8b80 0x7f9f6814c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 --2- 172.21.15.129:0/4136254367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f68072810 0x7f9f68141ec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 -- 172.21.15.129:0/4136254367 >> 172.21.15.129:0/4136254367 conn(0x7f9f6806aaf0 msgr2=0x7f9f680b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 -- 172.21.15.129:0/4136254367 shutdown_connections 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.658+0000 7f9f6d232640 1 -- 172.21.15.129:0/4136254367 wait complete. 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.659+0000 7f9f6d232640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.659+0000 7f9f6d232640 10 monclient: monmap: 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.700 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.701 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f677578c640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6770070210 0x7f677013e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f9f6d232640 1 librados: starting msgr at 2023-12-07T23:46:16.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f9f6d232640 1 librados: starting objecter 2023-12-07T23:46:16.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 1 -- 172.21.15.129:0/4214633297 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78030ad0 con 0x7ffb88139710 2023-12-07T23:46:16.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f677578c640 10 monclient(hunting): get_auth_request con 0x7f6770070210 auth_method 0 2023-12-07T23:46:16.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f677578c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f677578c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f677578c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f9f6d232640 1 Processor -- start 2023-12-07T23:46:16.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: dump: 2023-12-07T23:46:16.704 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 1 -- 172.21.15.129:0/4214633297 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb78030c30 con 0x7ffb88139710 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 1 -- 172.21.15.129:0/4214633297 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78038440 con 0x7ffb88139710 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb877fe640 10 monclient: dump: 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.707 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb8ec2d640 5 monclient: authenticate success, global_id 4317 2023-12-07T23:46:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f3796ffd640 1 --2- 172.21.15.129:0/688993352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900d01b0 0x7f3790030820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3784009870 tx=0x7f378402ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d60dc454eb90571 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb8ec2d640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7ffb8ec2d640 4 set_mon_vals no callback set 2023-12-07T23:46:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6774f8b640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67700aee70 0x7f67701da5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6774f8b640 10 monclient(hunting): get_auth_request con 0x7f67700aee70 auth_method 0 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6774f8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6774f8b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6774f8b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f677578c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f677578c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f677578c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f9f6d232640 1 -- start start 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f9f6d232640 1 librados: setting wanted keys 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f9f6d232640 1 librados: calling monclient init 2023-12-07T23:46:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.660+0000 7f9f6d232640 10 monclient: init 2023-12-07T23:46:16.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:16.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f68072430 0x7f9f6806efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 10 monclient: shutdown 2023-12-07T23:46:16.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6775f8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6775f8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f6775f8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): picked mon.noname-c con 0x7f9f68072430 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f680a8b80 0x7f9f681db800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/4214633297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 msgr2=0x7ffb88139ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): picked mon.noname-a con 0x7f9f680a8b80 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/4214633297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb88139ba0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb8806b330 tx=0x7ffb7802fd20 comp rx=0 tx=0).stop 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/4214633297 shutdown_connections 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/4214633297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb88139ba0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/4214633297 >> v2:172.21.15.184:3300/0 conn(0x7ffb88143840 0x7ffb881391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/4214633297 >> 172.21.15.129:0/4214633297 conn(0x7ffb880b03b0 msgr2=0x7ffb8806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/4214633297 shutdown_connections 2023-12-07T23:46:16.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/4214633297 wait complete. 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f681dbd40 0x7f9f6813a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): picked mon.noname-b con 0x7f9f681dbd40 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f681dc1d0 con 0x7f9f680a8b80 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680703a0 con 0x7f9f681dbd40 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f68070610 con 0x7f9f68072430 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.661+0000 7f9f6d232640 10 monclient(hunting): authenticate will time out at 1376.849879s 2023-12-07T23:46:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a557fa640 1 -- 172.21.15.129:0/3075773185 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a500078d0 con 0x7f4a58152780 2023-12-07T23:46:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a557fa640 10 monclient: dump: 2023-12-07T23:46:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a557fa640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a557fa640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 5 monclient: authenticate success, global_id 4268 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 10 monclient: _renew_subs 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a581e4650 con 0x7f4a58152780 2023-12-07T23:46:16.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 10 monclient: _renew_subs 2023-12-07T23:46:16.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a581e6f90 con 0x7f4a58152780 2023-12-07T23:46:16.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7ffb8ec2d640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7f4a5eaec640 1 librados: init done 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.662+0000 7ffb8ec2d640 10 monclient: monmap: 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.720 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7ffb8ec2d640 1 librados: starting msgr at 2023-12-07T23:46:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7ffb8ec2d640 1 librados: starting objecter 2023-12-07T23:46:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7ffb8ec2d640 1 Processor -- start 2023-12-07T23:46:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7f6774f8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7f6774f8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7f6774f8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7ffb8ec2d640 1 -- start start 2023-12-07T23:46:16.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7ffb8ec2d640 1 librados: setting wanted keys 2023-12-07T23:46:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7ffb8ec2d640 1 librados: calling monclient init 2023-12-07T23:46:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7ffb8ec2d640 10 monclient: init 2023-12-07T23:46:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.663+0000 7f9f6659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f680a8b80 0x7f9f681db800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7f4a5eaec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.724 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7f4a5eaec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.724 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7f4a5eaec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.724 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5808a2c0 con 0x7f4a58152780 2023-12-07T23:46:16.725 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb880aef00 0x7ffb881cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb880aef00 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb881d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb88139710 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb88143840 0x7ffb880aa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb88143840 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb881432b0 con 0x7ffb88139710 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb880a9f70 con 0x7ffb880aef00 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.664+0000 7ffb8ec2d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb880aa210 con 0x7ffb88143840 2023-12-07T23:46:16.727 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7ffb8ec2d640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.727 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7ffb8ec2d640 10 monclient(hunting): authenticate will time out at 1376.853047s 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 1 -- 172.21.15.129:0/732287447 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f677013d760 msgr2=0x7f677013dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f677013d760 0x7f677013dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 1 -- 172.21.15.129:0/732287447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67700aee70 msgr2=0x7f67701da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67700aee70 0x7f67701da5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.665+0000 7f677578c640 1 -- 172.21.15.129:0/732287447 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6760009400 con 0x7f6770070210 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.668+0000 7f9f6659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f680a8b80 0x7f9f681db800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42916/0 (socket says 172.21.15.129:42916) 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.668+0000 7f9f6659c640 1 -- 172.21.15.129:0/3367175026 learned_addr learned my addr 172.21.15.129:0/3367175026 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.668+0000 7f9f6659c640 10 monclient(hunting): get_auth_request con 0x7f9f680a8b80 auth_method 0 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.668+0000 7f9f6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.668+0000 7f9f6659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.668+0000 7f9f6659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.668+0000 7f9f6759e640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f681dbd40 0x7f9f6813a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.669+0000 7f677578c640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6770070210 0x7f677013e500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f676c00b6f0 tx=0x7f676c009e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.669+0000 7f6765ffb640 10 client.?.objecter ms_handle_connect 0x7f6770070210 2023-12-07T23:46:16.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.669+0000 7f6765ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.730 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.670+0000 7f9f6759e640 10 monclient(hunting): get_auth_request con 0x7f9f681dbd40 auth_method 0 2023-12-07T23:46:16.730 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.670+0000 7f9f6759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.730 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.670+0000 7f9f6759e640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.730 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.670+0000 7f9f6759e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.730 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.670+0000 7f9f6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.730 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.670+0000 7f9f6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.670+0000 7f9f6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.671+0000 7f9f6759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.671+0000 7f9f6759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.671+0000 7f9f6759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 1 -- 172.21.15.129:0/688993352 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378402f840 con 0x7f37900d01b0 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: dump: 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 1 -- 172.21.15.129:0/688993352 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378402f9a0 con 0x7f37900d01b0 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f379df16640 5 monclient: authenticate success, global_id 4336 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f379df16640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 1 -- 172.21.15.129:0/688993352 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378402fd30 con 0x7f37900d01b0 2023-12-07T23:46:16.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f379df16640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f379df16640 4 set_mon_vals no callback set 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.672+0000 7f37967fc640 10 monclient: dump: 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 10 monclient: shutdown 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 1 -- 172.21.15.129:0/688993352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900d01b0 msgr2=0x7f3790030820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 1 --2- 172.21.15.129:0/688993352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900d01b0 0x7f3790030820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3784009870 tx=0x7f378402ba20 comp rx=0 tx=0).stop 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 1 -- 172.21.15.129:0/688993352 shutdown_connections 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 1 --2- 172.21.15.129:0/688993352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900d01b0 0x7f3790030820 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 1 -- 172.21.15.129:0/688993352 >> 172.21.15.129:0/688993352 conn(0x7f3790006190 msgr2=0x7f37900065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 1 -- 172.21.15.129:0/688993352 shutdown_connections 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.673+0000 7f379df16640 1 -- 172.21.15.129:0/688993352 wait complete. 2023-12-07T23:46:16.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f4a557fa640 1 -- 172.21.15.129:0/3075773185 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50033020 con 0x7f4a58152780 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f4a557fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f4a56ffd640 4 set_mon_vals no callback set 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f4a557fa640 1 -- 172.21.15.129:0/3075773185 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5000a8a0 con 0x7f4a58152780 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f4a557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f4a557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f4a557fa640 10 monclient: dump: 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.738 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 10 monclient: monmap: 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 1 librados: starting msgr at 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 1 librados: starting objecter 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 1 Processor -- start 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 1 -- start start 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 1 librados: setting wanted keys 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 1 librados: calling monclient init 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.674+0000 7f379df16640 10 monclient: init 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 1 -- 172.21.15.129:0/1369892602 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda680317c0 con 0x7fda80143be0 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: dump: 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 1 -- 172.21.15.129:0/3367175026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f68072430 msgr2=0x7f9f6806efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f68072430 0x7f9f6806efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 1 -- 172.21.15.129:0/3367175026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f680a8b80 msgr2=0x7f9f681db800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f680a8b80 0x7f9f681db800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f9f6759e640 1 -- 172.21.15.129:0/3367175026 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f580094d0 con 0x7f9f681dbd40 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 1 -- 172.21.15.129:0/1369892602 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda68031920 con 0x7fda80143be0 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 1 -- 172.21.15.129:0/1369892602 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda680317c0 con 0x7fda80143be0 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda7dd9b640 10 monclient: dump: 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.743 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda84f00640 5 monclient: authenticate success, global_id 4265 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda84f00640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.744 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7fda84f00640 4 set_mon_vals no callback set 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900071c0 0x7f37900c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): picked mon.noname-b con 0x7f37900071c0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3790033550 0x7f37901655a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): picked mon.noname-a con 0x7f3790033550 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37900d01b0 0x7f3790165ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.745 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): picked mon.noname-c con 0x7f37900d01b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790166020 con 0x7f3790033550 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790165110 con 0x7f37900071c0 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37900c1aa0 con 0x7f37900d01b0 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.675+0000 7f379df16640 10 monclient(hunting): authenticate will time out at 1376.863655s 2023-12-07T23:46:16.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.676+0000 7fda84f00640 10 monclient: shutdown 2023-12-07T23:46:16.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.676+0000 7fda84f00640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.676+0000 7fda84f00640 1 -- 172.21.15.129:0/1369892602 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda80143be0 msgr2=0x7fda80070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.676+0000 7fda84f00640 1 --2- 172.21.15.129:0/1369892602 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda80143be0 0x7fda80070dc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda680098a0 tx=0x7fda6802b860 comp rx=0 tx=0).stop 2023-12-07T23:46:16.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.676+0000 7fda84f00640 1 -- 172.21.15.129:0/1369892602 shutdown_connections 2023-12-07T23:46:16.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.676+0000 7fda84f00640 1 --2- 172.21.15.129:0/1369892602 >> v2:172.21.15.184:3300/0 conn(0x7fda8006e970 0x7fda80071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.676+0000 7fda84f00640 1 --2- 172.21.15.129:0/1369892602 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda80143be0 0x7fda80070dc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.677+0000 7ffb8c9c9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb880aef00 0x7ffb881cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.677+0000 7fda84f00640 1 -- 172.21.15.129:0/1369892602 >> 172.21.15.129:0/1369892602 conn(0x7fda800b7710 msgr2=0x7fda800b7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.677+0000 7fda84f00640 1 -- 172.21.15.129:0/1369892602 shutdown_connections 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.677+0000 7fda84f00640 1 -- 172.21.15.129:0/1369892602 wait complete. 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.678+0000 7fda84f00640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.678+0000 7fda84f00640 10 monclient: monmap: 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.749 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.678+0000 7fda84f00640 1 librados: starting msgr at 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.678+0000 7fda84f00640 1 librados: starting objecter 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.678+0000 7fda84f00640 1 Processor -- start 2023-12-07T23:46:16.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb87fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb881d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 1 -- start start 2023-12-07T23:46:16.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 1 librados: setting wanted keys 2023-12-07T23:46:16.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 1 librados: calling monclient init 2023-12-07T23:46:16.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 10 monclient: init 2023-12-07T23:46:16.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb87fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb881d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42928/0 (socket says 172.21.15.129:42928) 2023-12-07T23:46:16.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb87fff640 1 -- 172.21.15.129:0/2628337485 learned_addr learned my addr 172.21.15.129:0/2628337485 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb87fff640 10 monclient(hunting): get_auth_request con 0x7ffb88139710 auth_method 0 2023-12-07T23:46:16.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb87fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb87fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7ffb8d1ca640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb88143840 0x7ffb880aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda8006e970 0x7fda80139580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 10 monclient(hunting): picked mon.noname-c con 0x7fda8006e970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.679+0000 7fda84f00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda800b3390 0x7fda80140670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 10 monclient(hunting): picked mon.noname-a con 0x7fda800b3390 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda80143be0 0x7fda80146330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 10 monclient(hunting): picked mon.noname-b con 0x7fda80143be0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7ffb87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7ffb87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7ffb87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80144140 con 0x7fda800b3390 2023-12-07T23:46:16.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80144d70 con 0x7fda80143be0 2023-12-07T23:46:16.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda80140c30 con 0x7fda8006e970 2023-12-07T23:46:16.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7fda84f00640 10 monclient(hunting): authenticate will time out at 1376.868254s 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7f9f6759e640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f681dbd40 0x7f9f6813a3f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00d490 tx=0x7f9f5c00d950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7f9f4f7fe640 10 client.?.objecter ms_handle_connect 0x7f9f681dbd40 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.680+0000 7f9f4f7fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.681+0000 7ffb8c9c9640 10 monclient(hunting): get_auth_request con 0x7ffb880aef00 auth_method 0 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.681+0000 7ffb8c9c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.681+0000 7ffb8c9c9640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.681+0000 7ffb8c9c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.681+0000 7f37977fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900071c0 0x7f37900c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7ffb8c9c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7ffb8c9c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7ffb8c9c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7f37977fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900071c0 0x7f37900c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40672/0 (socket says 172.21.15.129:40672) 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7f37977fe640 1 -- 172.21.15.129:0/2040473104 learned_addr learned my addr 172.21.15.129:0/2040473104 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7f37977fe640 10 monclient(hunting): get_auth_request con 0x7f37900071c0 auth_method 0 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7f37977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7f37977fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.682+0000 7f37977fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 1 -- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb88143840 msgr2=0x7ffb880aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb88143840 0x7ffb880aa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 1 -- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb880aef00 msgr2=0x7ffb881cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb880aef00 0x7ffb881cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7ffb87fff640 1 -- 172.21.15.129:0/2628337485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780094d0 con 0x7ffb88139710 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.683+0000 7f3796ffd640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3790033550 0x7f37901655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.684+0000 7f3796ffd640 10 monclient(hunting): get_auth_request con 0x7f3790033550 auth_method 0 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.684+0000 7f3796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.684+0000 7f3796ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.684+0000 7f3796ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.684+0000 7f3796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.684+0000 7f3796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.684+0000 7f3796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.685+0000 7f37977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.685+0000 7f37977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.685+0000 7f37977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2023-12-07T23:46:16.761 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 1 -- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37900d01b0 msgr2=0x7f3790165ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37900d01b0 0x7f3790165ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 1 -- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900071c0 msgr2=0x7f37900c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900071c0 0x7f37900c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.689+0000 7f3796ffd640 1 -- 172.21.15.129:0/2040473104 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840094d0 con 0x7f3790033550 2023-12-07T23:46:16.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7fda7ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda80143be0 0x7fda80146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 1 -- 172.21.15.129:0/3367175026 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c018070 con 0x7f9f681dbd40 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: dump: 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.763 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 1 -- 172.21.15.129:0/3367175026 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c004020 con 0x7f9f681dbd40 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 1 -- 172.21.15.129:0/3367175026 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c0041b0 con 0x7f9f681dbd40 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f4f7fe640 10 monclient: dump: 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7ffb87fff640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb881d04f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00b3f0 tx=0x7ffb7c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7ffb857fa640 10 client.?.objecter ms_handle_connect 0x7ffb88139710 2023-12-07T23:46:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7ffb857fa640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 5 monclient: authenticate success, global_id 4345 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 10 monclient: _renew_subs 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6806e220 con 0x7f9f681dbd40 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 10 monclient: _renew_subs 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6813b040 con 0x7f9f681dbd40 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6d232640 1 librados: init done 2023-12-07T23:46:16.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.692+0000 7f9f6559a640 4 set_mon_vals no callback set 2023-12-07T23:46:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7fda7ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda80143be0 0x7fda80146330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40678/0 (socket says 172.21.15.129:40678) 2023-12-07T23:46:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7fda7ed9d640 1 -- 172.21.15.129:0/1278061754 learned_addr learned my addr 172.21.15.129:0/1278061754 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7fda7ed9d640 10 monclient(hunting): get_auth_request con 0x7fda80143be0 auth_method 0 2023-12-07T23:46:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7fda7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7fda7ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.693+0000 7fda7ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7fda7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 1 -- 172.21.15.129:0/732287447 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c010040 con 0x7f6770070210 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: dump: 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 1 -- 172.21.15.129:0/732287447 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c00eb70 con 0x7f6770070210 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 1 -- 172.21.15.129:0/732287447 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c016410 con 0x7f6770070210 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f6765ffb640 10 monclient: dump: 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.769 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.770 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f9f6d232640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f9f6d232640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f9f6d232640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7fda7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7fda7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.771 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f681db520 con 0x7f9f681dbd40 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f67677fe640 4 set_mon_vals no callback set 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 5 monclient: authenticate success, global_id 4342 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 10 monclient: _renew_subs 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67701dab00 con 0x7f6770070210 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 10 monclient: _renew_subs 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67701cb6f0 con 0x7f6770070210 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.694+0000 7f677678e640 1 librados: init done 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.696+0000 7f677678e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.696+0000 7f677678e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.696+0000 7f677678e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:16.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.696+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67701db130 con 0x7f6770070210 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 1 -- 172.21.15.129:0/2628337485 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c005970 con 0x7ffb88139710 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: dump: 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.773 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 1 -- 172.21.15.129:0/2628337485 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c005ad0 con 0x7ffb88139710 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb8ec2d640 5 monclient: authenticate success, global_id 4329 2023-12-07T23:46:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 1 -- 172.21.15.129:0/2628337485 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c005e80 con 0x7ffb88139710 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb857fa640 10 monclient: dump: 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7ffb86ffd640 4 set_mon_vals no callback set 2023-12-07T23:46:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.698+0000 7fda77fff640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda800b3390 0x7fda80140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7f3796ffd640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3790033550 0x7f37901655a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37900344e0 tx=0x7f37840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7f3783fff640 10 client.?.objecter ms_handle_connect 0x7f3790033550 2023-12-07T23:46:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7f3783fff640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 10 monclient: _renew_subs 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb88143e50 con 0x7ffb88139710 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 10 monclient: _renew_subs 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb880aaf80 con 0x7ffb88139710 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.699+0000 7ffb8ec2d640 1 librados: init done 2023-12-07T23:46:16.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7ffb8ec2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7fda77fff640 10 monclient(hunting): get_auth_request con 0x7fda800b3390 auth_method 0 2023-12-07T23:46:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7fda77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7fda77fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7fda77fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7ffb8ec2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7ffb8ec2d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.700+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb880b2f00 con 0x7ffb88139710 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.701+0000 7fc16a64b640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.701+0000 7fc16a64b640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.701+0000 7fc16a64b640 10 monclient: monmap: 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.703062+0000 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.703062+0000 2023-12-07T23:46:16.779 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.702+0000 7fc16a64b640 1 Processor -- start 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.702+0000 7fc16a64b640 1 -- start start 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.702+0000 7fc16a64b640 10 monclient: init 2023-12-07T23:46:16.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.702+0000 7f5afe3f0640 10 monclient: get_monmap_and_config 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.702+0000 7f5afe3f0640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.702+0000 7f5afe3f0640 10 monclient: monmap: 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:16.703907+0000 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:16.703907+0000 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:16.781 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7f5afe3f0640 1 Processor -- start 2023-12-07T23:46:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-12-07T23:46:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc1640a8970 0x7fc16413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1640a8970 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:16.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc1640a9bd0 0x7fc16413f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7f5afe3f0640 1 -- start start 2023-12-07T23:46:16.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7f5afe3f0640 10 monclient: init 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1640a9bd0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1641421a0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640aa950 con 0x7fc1641421a0 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1640a90e0 con 0x7fc1640a9bd0 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1640aabc0 con 0x7fc1640a8970 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.703+0000 7fc16a64b640 10 monclient(hunting): authenticate will time out at 1376.891751s 2023-12-07T23:46:16.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-12-07T23:46:16.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient(hunting): picked mon.noname-b con 0x7f5af80a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5af80a9bd0 0x7f5af813f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient(hunting): picked mon.noname-a con 0x7f5af80a9bd0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5af81421a0 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af80aa950 con 0x7f5af81421a0 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af80a90e0 con 0x7f5af80a9bd0 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af8142d40 con 0x7f5af80a8970 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.704+0000 7f5afe3f0640 10 monclient(hunting): authenticate will time out at 1376.892483s 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.709+0000 7fda77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.709+0000 7fda77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.709+0000 7fda77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 1 -- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda8006e970 msgr2=0x7fda80139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda8006e970 0x7fda80139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 1 -- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda80143be0 msgr2=0x7fda80146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda80143be0 0x7fda80146330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 1 monclient: found mon.noname-a 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.716+0000 7fda77fff640 1 -- 172.21.15.129:0/1278061754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda680094d0 con 0x7fda800b3390 2023-12-07T23:46:16.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.729+0000 7ffb857fa640 1 -- 172.21.15.129:0/2628337485 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffb7c011630 con 0x7ffb88139710 2023-12-07T23:46:16.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.730+0000 7ffb857fa640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb7406c820 0x7ffb7406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.730+0000 7ffb8c9c9640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb7406c820 0x7ffb7406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.730+0000 7ffb8c9c9640 10 monclient: get_auth_request con 0x7ffb7406c820 auth_method 0 2023-12-07T23:46:16.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.731+0000 7ffb8c9c9640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb7406c820 0x7ffb7406ece0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffb80001ef0 tx=0x7ffb80009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.731+0000 7ffb857fa640 10 client.4329.objecter ms_handle_connect 0x7ffb7406c820 2023-12-07T23:46:16.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 1 -- 172.21.15.129:0/2040473104 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378403b040 con 0x7f3790033550 2023-12-07T23:46:16.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:16.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 10 monclient: dump: 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.923 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 1 -- 172.21.15.129:0/2040473104 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3784037e30 con 0x7f3790033550 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 5 monclient: authenticate success, global_id 4332 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 10 monclient: _renew_subs 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37900c3c80 con 0x7f3790033550 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3783fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f3795ffb640 4 set_mon_vals no callback set 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 10 monclient: _renew_subs 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37900c4950 con 0x7f3790033550 2023-12-07T23:46:16.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.919+0000 7f379df16640 1 librados: init done 2023-12-07T23:46:16.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.920+0000 7f379df16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.920+0000 7f379df16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.920+0000 7f379df16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:16.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.920+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790008090 con 0x7f3790033550 2023-12-07T23:46:16.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.934+0000 7f4a557fa640 1 -- 172.21.15.129:0/3075773185 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4a50054430 con 0x7f4a58152780 2023-12-07T23:46:16.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.934+0000 7f4a557fa640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a4406c850 0x7f4a4406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.935+0000 7f4a5c888640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a4406c850 0x7f4a4406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.935+0000 7f4a5c888640 10 monclient: get_auth_request con 0x7f4a4406c850 auth_method 0 2023-12-07T23:46:16.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.935+0000 7f4a557fa640 10 client.4268.objecter ms_handle_connect 0x7f4a4406c850 2023-12-07T23:46:16.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.935+0000 7f4a5c888640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a4406c850 0x7f4a4406ed10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a480011e0 tx=0x7f4a48009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 1 -- 172.21.15.129:0/3075773185 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4a5007fd70 con 0x7f4a58152780 2023-12-07T23:46:16.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 10 client.4268.objecter ms_dispatch 0x7f4a58000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 3 client.4268.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 3 client.4268.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 20 client.4268.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 1 -- 172.21.15.129:0/3075773185 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a5007b9b0 con 0x7f4a58152780 2023-12-07T23:46:16.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.947+0000 7f4a557fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:16.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.948+0000 7f5af7fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.948+0000 7f5af7fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55658/0 (socket says 172.21.15.129:55658) 2023-12-07T23:46:16.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.948+0000 7f5af7fff640 1 -- 172.21.15.129:0/232215078 learned_addr learned my addr 172.21.15.129:0/232215078 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.948+0000 7f5af7fff640 10 monclient(hunting): get_auth_request con 0x7f5af80a8970 auth_method 0 2023-12-07T23:46:16.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.948+0000 7f5af7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.948+0000 7f5af7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.948+0000 7f5af7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.949+0000 7f5af7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.949+0000 7f5af7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.949+0000 7f5af7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 1 -- 172.21.15.129:0/232215078 >> v2:172.21.15.129:3300/0 conn(0x7f5af80a9bd0 msgr2=0x7f5af813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 1 --2- 172.21.15.129:0/232215078 >> v2:172.21.15.129:3300/0 conn(0x7f5af80a9bd0 0x7f5af813f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 1 -- 172.21.15.129:0/232215078 >> v1:172.21.15.184:6789/0 conn(0x7f5af81421a0 legacy=0x7f5af8142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 1 monclient: found mon.noname-b 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.952+0000 7f5af7fff640 1 -- 172.21.15.129:0/232215078 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af80aabc0 con 0x7f5af80a8970 2023-12-07T23:46:16.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.954+0000 7f5af7fff640 1 --2- 172.21.15.129:0/232215078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813ee70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009870 tx=0x7f5ae802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ec0081827e7e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 1 -- 172.21.15.129:0/232215078 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8031740 con 0x7f5af80a8970 2023-12-07T23:46:16.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:16.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: dump: 2023-12-07T23:46:16.961 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 1 -- 172.21.15.129:0/232215078 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae80318a0 con 0x7f5af80a8970 2023-12-07T23:46:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5afe3f0640 5 monclient: authenticate success, global_id 4274 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5afe3f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5afe3f0640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5afe3f0640 4 set_mon_vals no callback set 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 1 -- 172.21.15.129:0/232215078 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8031c30 con 0x7f5af80a8970 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.958+0000 7f5af6ffd640 10 monclient: dump: 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.963 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.964 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.964 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.964 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.966 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 10 monclient: shutdown 2023-12-07T23:46:16.966 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:16.966 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 -- 172.21.15.129:0/232215078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 msgr2=0x7f5af813ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.966 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/232215078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813ee70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009870 tx=0x7f5ae802b7e0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.966 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 -- 172.21.15.129:0/232215078 shutdown_connections 2023-12-07T23:46:16.966 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/232215078 >> v2:172.21.15.129:3300/0 conn(0x7f5af80a9bd0 0x7f5af813f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.966 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/232215078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813ee70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 -- 172.21.15.129:0/232215078 >> 172.21.15.129:0/232215078 conn(0x7f5af8066dd0 msgr2=0x7f5af80b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 -- 172.21.15.129:0/232215078 shutdown_connections 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.959+0000 7f5afe3f0640 1 -- 172.21.15.129:0/232215078 wait complete. 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 10 monclient: build_initial_monmap 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 10 monclient: monmap: 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:16.967 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 1 librados: starting msgr at 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 1 librados: starting objecter 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 1 Processor -- start 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 1 -- start start 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 1 librados: setting wanted keys 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 1 librados: calling monclient init 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.960+0000 7f5afe3f0640 10 monclient: init 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): picked mon.noname-c con 0x7f5af80a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5af80a9bd0 0x7f5af80ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): picked mon.noname-b con 0x7f5af80a9bd0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5af81421a0 0x7f5af80b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): picked mon.noname-a con 0x7f5af81421a0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:16.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80aef40 con 0x7f5af81421a0 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80ada30 con 0x7f5af80a9bd0 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80acfc0 con 0x7f5af80a8970 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5afe3f0640 10 monclient(hunting): authenticate will time out at 1377.149383s 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.961+0000 7f5af7fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55664/0 (socket says 172.21.15.129:55664) 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 1 -- 172.21.15.129:0/2689208435 learned_addr learned my addr 172.21.15.129:0/2689208435 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 10 monclient(hunting): get_auth_request con 0x7f5af80a8970 auth_method 0 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.962+0000 7f5af7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.963+0000 7f3783fff640 1 -- 172.21.15.129:0/2040473104 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784042680 con 0x7f3790033550 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.963+0000 7f3783fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.963+0000 7f3783fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.963+0000 7f3783fff640 10 monclient: dump: 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.970 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.971 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.971 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.971 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.971 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.971 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.971 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 1 -- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5af80a9bd0 msgr2=0x7f5af80ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5af80a9bd0 0x7f5af80ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 1 -- 172.21.15.129:0/2689208435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5af81421a0 msgr2=0x7f5af80b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5af81421a0 0x7f5af80b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 1 monclient: found mon.noname-c 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.965+0000 7f5af7fff640 1 -- 172.21.15.129:0/2689208435 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae80094d0 con 0x7f5af80a8970 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.967+0000 7f5af7fff640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813a760 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009870 tx=0x7f5ae80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.967+0000 7f5af4ff9640 10 client.?.objecter ms_handle_connect 0x7f5af80a8970 2023-12-07T23:46:16.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.967+0000 7f5af4ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:16.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 1 -- 172.21.15.129:0/2689208435 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8033c70 con 0x7f5af80a8970 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: dump: 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: _finish_auth 0 2023-12-07T23:46:16.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 1 -- 172.21.15.129:0/2689208435 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8033dd0 con 0x7f5af80a8970 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 5 monclient: authenticate success, global_id 4277 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 10 monclient: _renew_subs 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af80ad180 con 0x7f5af80a8970 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 1 -- 172.21.15.129:0/2689208435 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae803d6b0 con 0x7f5af80a8970 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af4ff9640 10 monclient: dump: 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.976 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 10 monclient: _renew_subs 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af80b0f50 con 0x7f5af80a8970 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5af67fc640 4 set_mon_vals no callback set 2023-12-07T23:46:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.971+0000 7f5afe3f0640 1 librados: init done 2023-12-07T23:46:16.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.972+0000 7f5afe3f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.972+0000 7f5afe3f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.972+0000 7f5afe3f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:16.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.972+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af804e840 con 0x7f5af80a8970 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 1 -- 172.21.15.129:0/2628337485 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffb7c058f40 con 0x7ffb88139710 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 10 client.4329.objecter ms_dispatch 0x7ffb88000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 3 client.4329.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 3 client.4329.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 20 client.4329.objecter dump_active .. 0 homeless 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 1 -- 172.21.15.129:0/2628337485 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb7c054b80 con 0x7ffb88139710 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.975+0000 7ffb857fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.977+0000 7fda77fff640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda800b3390 0x7fda80140670 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fda68033000 tx=0x7fda680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:16.979 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.977+0000 7fda76ffd640 10 client.?.objecter ms_handle_connect 0x7fda800b3390 2023-12-07T23:46:16.980 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:16.977+0000 7fda76ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:17.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.158+0000 7f4a5eaec640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:17.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.158+0000 7f4a5eaec640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:17.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.158+0000 7f4a5eaec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:17.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.158+0000 7f9f4f7fe640 1 -- 172.21.15.129:0/3367175026 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9f5c004310 con 0x7f9f681dbd40 2023-12-07T23:46:17.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.158+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4a5806fed0 con 0x7f4a58152780 2023-12-07T23:46:17.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.159+0000 7f9f4f7fe640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9f5406c820 0x7f9f5406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.159+0000 7f9f66d9d640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9f5406c820 0x7f9f5406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.159+0000 7f9f66d9d640 10 monclient: get_auth_request con 0x7f9f5406c820 auth_method 0 2023-12-07T23:46:17.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.159+0000 7f9f4f7fe640 10 client.4345.objecter ms_handle_connect 0x7f9f5406c820 2023-12-07T23:46:17.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.159+0000 7f9f66d9d640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9f5406c820 0x7f9f5406ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f600012e0 tx=0x7f9f60007d30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.167+0000 7ffb8ec2d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:17.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.167+0000 7ffb8ec2d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:17.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.167+0000 7ffb8ec2d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:17.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.167+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffb88071e10 con 0x7ffb88139710 2023-12-07T23:46:17.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.169+0000 7f6765ffb640 1 -- 172.21.15.129:0/732287447 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f676c016570 con 0x7f6770070210 2023-12-07T23:46:17.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.170+0000 7f6765ffb640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f675406c870 0x7f675406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.170+0000 7f6774f8b640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f675406c870 0x7f675406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.170+0000 7f6774f8b640 10 monclient: get_auth_request con 0x7f675406c870 auth_method 0 2023-12-07T23:46:17.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.170+0000 7f6774f8b640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f675406c870 0x7f675406ed30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67701ca6b0 tx=0x7f676800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.171+0000 7f6765ffb640 10 client.4342.objecter ms_handle_connect 0x7f675406c870 2023-12-07T23:46:17.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.173+0000 7f9f4f7fe640 1 -- 172.21.15.129:0/3367175026 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9f5c0590a0 con 0x7f9f681dbd40 2023-12-07T23:46:17.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.173+0000 7f9f4f7fe640 10 client.4345.objecter ms_dispatch 0x7f9f68000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:17.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.173+0000 7f9f4f7fe640 3 client.4345.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:17.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.173+0000 7f9f4f7fe640 3 client.4345.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:17.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.174+0000 7f9f4f7fe640 20 client.4345.objecter dump_active .. 0 homeless 2023-12-07T23:46:17.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.186+0000 7f9f4f7fe640 1 -- 172.21.15.129:0/3367175026 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f5c054ce0 con 0x7f9f681dbd40 2023-12-07T23:46:17.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.186+0000 7f9f4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.186+0000 7f9f4f7fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:17.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.190+0000 7f6765ffb640 1 -- 172.21.15.129:0/732287447 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f676c058d80 con 0x7f6770070210 2023-12-07T23:46:17.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.190+0000 7f6765ffb640 10 client.4342.objecter ms_dispatch 0x7f6770000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:17.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.190+0000 7f6765ffb640 3 client.4342.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:17.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.190+0000 7f6765ffb640 3 client.4342.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:17.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.190+0000 7f6765ffb640 20 client.4342.objecter dump_active .. 0 homeless 2023-12-07T23:46:17.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.200+0000 7f6765ffb640 1 -- 172.21.15.129:0/732287447 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f676c0550c0 con 0x7f6770070210 2023-12-07T23:46:17.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.200+0000 7f6765ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.200+0000 7f6765ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:17.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.203+0000 7fc163fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc1640a8970 0x7fc16413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.205+0000 7fc163fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc1640a8970 0x7fc16413ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40688/0 (socket says 172.21.15.129:40688) 2023-12-07T23:46:17.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.205+0000 7fc163fff640 1 -- 172.21.15.129:0/3902029613 learned_addr learned my addr 172.21.15.129:0/3902029613 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:17.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.205+0000 7fc163fff640 10 monclient(hunting): get_auth_request con 0x7fc1640a8970 auth_method 0 2023-12-07T23:46:17.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.205+0000 7fc163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:17.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.205+0000 7fc163fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:17.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.205+0000 7fc163fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:17.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.206+0000 7fc163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:17.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.206+0000 7fc163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:17.210 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.207+0000 7fc163fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:17.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2023-12-07T23:46:17.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 1 -- 172.21.15.129:0/3902029613 >> v2:172.21.15.129:3300/0 conn(0x7fc1640a9bd0 msgr2=0x7fc16413f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:17.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 1 --2- 172.21.15.129:0/3902029613 >> v2:172.21.15.129:3300/0 conn(0x7fc1640a9bd0 0x7fc16413f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 1 -- 172.21.15.129:0/3902029613 >> v1:172.21.15.129:6789/0 conn(0x7fc1641421a0 legacy=0x7fc1641425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:17.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:17.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 1 monclient: found mon.noname-c 2023-12-07T23:46:17.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:17.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.209+0000 7fc163fff640 1 -- 172.21.15.129:0/3902029613 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1640aaea0 con 0x7fc1640a8970 2023-12-07T23:46:17.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.212+0000 7fc163fff640 1 --2- 172.21.15.129:0/3902029613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc16413ee70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc1640b4f90 tx=0x7fc15402b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a794094f319a959 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.218 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 1 -- 172.21.15.129:0/3902029613 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154031750 con 0x7fc1640a8970 2023-12-07T23:46:17.218 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: dump: 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:17.219 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 1 -- 172.21.15.129:0/3902029613 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1540318b0 con 0x7fc1640a8970 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc16a64b640 5 monclient: authenticate success, global_id 4357 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc16a64b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc16a64b640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc16a64b640 4 set_mon_vals no callback set 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 1 -- 172.21.15.129:0/3902029613 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154031c40 con 0x7fc1640a8970 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.215+0000 7fc162ffd640 10 monclient: dump: 2023-12-07T23:46:17.220 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:17.221 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 10 monclient: shutdown 2023-12-07T23:46:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 -- 172.21.15.129:0/3902029613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 msgr2=0x7fc16413ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 --2- 172.21.15.129:0/3902029613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc16413ee70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc1640b4f90 tx=0x7fc15402b7f0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 -- 172.21.15.129:0/3902029613 shutdown_connections 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 --2- 172.21.15.129:0/3902029613 >> v2:172.21.15.129:3300/0 conn(0x7fc1640a9bd0 0x7fc16413f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 --2- 172.21.15.129:0/3902029613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc16413ee70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 -- 172.21.15.129:0/3902029613 >> 172.21.15.129:0/3902029613 conn(0x7fc164066dd0 msgr2=0x7fc1640b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 -- 172.21.15.129:0/3902029613 shutdown_connections 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.216+0000 7fc16a64b640 1 -- 172.21.15.129:0/3902029613 wait complete. 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.217+0000 7fc16a64b640 10 monclient: build_initial_monmap 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.217+0000 7fc16a64b640 10 monclient: monmap: 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.218+0000 7fc16a64b640 1 librados: starting msgr at 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.218+0000 7fc16a64b640 1 librados: starting objecter 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.218+0000 7fc16a64b640 1 Processor -- start 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.218+0000 7fc16a64b640 1 -- start start 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.218+0000 7fc16a64b640 1 librados: setting wanted keys 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.218+0000 7fc16a64b640 1 librados: calling monclient init 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.218+0000 7fc16a64b640 10 monclient: init 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:17.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc1641daab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1640a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1640a9bd0 0x7fc164133950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1640a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1641421a0 0x7fc164133e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1641421a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1641da620 con 0x7fc1640a9bd0 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc164139e40 con 0x7fc1640a8970 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16413a000 con 0x7fc1641421a0 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.219+0000 7fc16a64b640 10 monclient(hunting): authenticate will time out at 1377.407855s 2023-12-07T23:46:17.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.220+0000 7fc163fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc1641daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.220+0000 7fc163fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc1641daab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40718/0 (socket says 172.21.15.129:40718) 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.220+0000 7fc163fff640 1 -- 172.21.15.129:0/4129620774 learned_addr learned my addr 172.21.15.129:0/4129620774 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.220+0000 7fc163fff640 10 monclient(hunting): get_auth_request con 0x7fc1640a8970 auth_method 0 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.220+0000 7fc163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.220+0000 7fc163fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.220+0000 7fc163fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.221+0000 7fc163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.221+0000 7fc163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.221+0000 7fc163fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 1 -- 172.21.15.129:0/4129620774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1641421a0 msgr2=0x7fc164133e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1641421a0 0x7fc164133e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 1 -- 172.21.15.129:0/4129620774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1640a9bd0 msgr2=0x7fc164133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1640a9bd0 0x7fc164133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 1 monclient: found mon.noname-b 2023-12-07T23:46:17.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:17.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.223+0000 7fc163fff640 1 -- 172.21.15.129:0/4129620774 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540094d0 con 0x7fc1640a8970 2023-12-07T23:46:17.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.226+0000 7fc163fff640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc1641daab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc1540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.226+0000 7fc160ff9640 10 client.?.objecter ms_handle_connect 0x7fc1640a8970 2023-12-07T23:46:17.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.226+0000 7fc160ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 1 -- 172.21.15.129:0/4129620774 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154033c60 con 0x7fc1640a8970 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: dump: 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.232 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: _finish_auth 0 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 1 -- 172.21.15.129:0/4129620774 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc154033de0 con 0x7fc1640a8970 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc16a64b640 5 monclient: authenticate success, global_id 4360 2023-12-07T23:46:17.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 1 -- 172.21.15.129:0/4129620774 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15403d6b0 con 0x7fc1640a8970 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc160ff9640 10 monclient: dump: 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:17.234 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:17.235 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:17.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.229+0000 7fc1627fc640 4 set_mon_vals no callback set 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 monclient: _renew_subs 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc164137970 con 0x7fc1640a8970 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 monclient: _renew_subs 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1641daff0 con 0x7fc1640a8970 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 1 librados: init done 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.230+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1641da3f0 con 0x7fc1640a8970 2023-12-07T23:46:17.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.249+0000 7fc160ff9640 1 -- 172.21.15.129:0/4129620774 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc15403a040 con 0x7fc1640a8970 2023-12-07T23:46:17.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.250+0000 7fc160ff9640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc15006c870 0x7fc15006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.250+0000 7fc1637fe640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc15006c870 0x7fc15006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.250+0000 7fc1637fe640 10 monclient: get_auth_request con 0x7fc15006c870 auth_method 0 2023-12-07T23:46:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.250+0000 7fc1637fe640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc15006c870 0x7fc15006ed30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1640abb00 tx=0x7fc158006c90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.250+0000 7fc160ff9640 10 client.4360.objecter ms_handle_connect 0x7fc15006c870 2023-12-07T23:46:17.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.251+0000 7f5af4ff9640 1 -- 172.21.15.129:0/2689208435 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5ae803a040 con 0x7f5af80a8970 2023-12-07T23:46:17.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.251+0000 7f5af4ff9640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5ae406c870 0x7f5ae406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.251+0000 7f5af77fe640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5ae406c870 0x7f5ae406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.252+0000 7f5af77fe640 10 monclient: get_auth_request con 0x7f5ae406c870 auth_method 0 2023-12-07T23:46:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.252+0000 7f5af4ff9640 10 client.4277.objecter ms_handle_connect 0x7f5ae406c870 2023-12-07T23:46:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.252+0000 7f5af77fe640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5ae406c870 0x7f5ae406ed30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5af813b890 tx=0x7f5aec006c90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.253+0000 7fc160ff9640 1 -- 172.21.15.129:0/4129620774 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc15407ae70 con 0x7fc1640a8970 2023-12-07T23:46:17.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.253+0000 7fc160ff9640 10 client.4360.objecter ms_dispatch 0x7fc164000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:17.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.253+0000 7fc160ff9640 3 client.4360.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:17.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.253+0000 7fc160ff9640 3 client.4360.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:17.256 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.253+0000 7fc160ff9640 20 client.4360.objecter dump_active .. 0 homeless 2023-12-07T23:46:17.268 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 1 -- 172.21.15.129:0/2689208435 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5ae803be40 con 0x7f5af80a8970 2023-12-07T23:46:17.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 10 client.4277.objecter ms_dispatch 0x7f5af8000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:17.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 3 client.4277.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:17.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 3 client.4277.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:17.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 20 client.4277.objecter dump_active .. 0 homeless 2023-12-07T23:46:17.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 1 -- 172.21.15.129:0/2689208435 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ae8076bb0 con 0x7f5af80a8970 2023-12-07T23:46:17.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.265+0000 7f5af4ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:17.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.273+0000 7f4a557fa640 1 -- 172.21.15.129:0/3075773185 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a5003c020 con 0x7f4a58152780 2023-12-07T23:46:17.276 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.273+0000 7f4a557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:17.277 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.273+0000 7f4a557fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.277 INFO:teuthology.orchestra.run.smithi129.stdout:55834574852 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 10 librados: watch_flush enter 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 10 librados: watch_flush exit 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 20 client.4268.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 10 client.4268.objecter successfully canceled tick 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a4406c850 msgr2=0x7f4a4406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a4406c850 0x7f4a4406ed10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a480011e0 tx=0x7f4a48009040 comp rx=0 tx=0).stop 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 10 monclient: shutdown 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 msgr2=0x7f4a5807ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.276+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a5807ad90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c430 tx=0x7f4a50004a40 comp rx=0 tx=0).stop 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 shutdown_connections 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f4a4406c850 0x7f4a4406ed10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a5807b2d0 0x7f4a581e4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4a58152780 0x7f4a5807ad90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 --2- 172.21.15.129:0/3075773185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a580b0980 0x7f4a58079ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 >> 172.21.15.129:0/3075773185 conn(0x7f4a5806aaf0 msgr2=0x7f4a580ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 shutdown_connections 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 -- 172.21.15.129:0/3075773185 wait complete. 2023-12-07T23:46:17.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.277+0000 7f4a5eaec640 1 librados: shutdown 2023-12-07T23:46:17.292 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2023-12-07T23:46:17.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.292+0000 7f3783fff640 1 -- 172.21.15.129:0/2040473104 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f37840427e0 con 0x7f3790033550 2023-12-07T23:46:17.295 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.293+0000 7f3783fff640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f377c06c870 0x7f377c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.293+0000 7f37977fe640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f377c06c870 0x7f377c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.293+0000 7f37977fe640 10 monclient: get_auth_request con 0x7f377c06c870 auth_method 0 2023-12-07T23:46:17.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.293+0000 7f37977fe640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f377c06c870 0x7f377c06ed30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3798068540 tx=0x7f3798068400 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.296 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.293+0000 7f3783fff640 10 client.4332.objecter ms_handle_connect 0x7f377c06c870 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.319+0000 7f3783fff640 1 -- 172.21.15.129:0/2040473104 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f378407c540 con 0x7f3790033550 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.319+0000 7f3783fff640 10 client.4332.objecter ms_dispatch 0x7f3798000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.319+0000 7f3783fff640 3 client.4332.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.319+0000 7f3783fff640 3 client.4332.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.320+0000 7f3783fff640 20 client.4332.objecter dump_active .. 0 homeless 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.320+0000 7f3783fff640 1 -- 172.21.15.129:0/2040473104 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3784078180 con 0x7f3790033550 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.320+0000 7f3783fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.320+0000 7f3783fff640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:17.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 1 -- 172.21.15.129:0/1278061754 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda680317c0 con 0x7fda800b3390 2023-12-07T23:46:17.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: dump: 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:17.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 1 -- 172.21.15.129:0/1278061754 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda68032920 con 0x7fda800b3390 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 5 monclient: authenticate success, global_id 4335 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 10 monclient: _renew_subs 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda80140e50 con 0x7fda800b3390 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 1 -- 172.21.15.129:0/1278061754 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68032cd0 con 0x7fda800b3390 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda7d59a640 4 set_mon_vals no callback set 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda76ffd640 10 monclient: dump: 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.351 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 10 monclient: _renew_subs 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.345+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda80146a50 con 0x7fda800b3390 2023-12-07T23:46:17.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.346+0000 7fda84f00640 1 librados: init done 2023-12-07T23:46:17.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.347+0000 7fda84f00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.347+0000 7fda84f00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.347+0000 7fda84f00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:17.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.347+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8006fbf0 con 0x7fda800b3390 2023-12-07T23:46:17.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.347+0000 7ffb857fa640 1 -- 172.21.15.129:0/2628337485 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffb7c015070 con 0x7ffb88139710 2023-12-07T23:46:17.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.347+0000 7ffb857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:17.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.347+0000 7ffb857fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.354 INFO:teuthology.orchestra.run.smithi129.stdout:68719476741 2023-12-07T23:46:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 10 librados: watch_flush enter 2023-12-07T23:46:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 10 librados: watch_flush exit 2023-12-07T23:46:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 20 client.4329.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 10 client.4329.objecter successfully canceled tick 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb7406c820 msgr2=0x7ffb7406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb7406c820 0x7ffb7406ece0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffb80001ef0 tx=0x7ffb80009040 comp rx=0 tx=0).stop 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 10 monclient: shutdown 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 msgr2=0x7ffb881d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb881d04f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00b3f0 tx=0x7ffb7c00b8b0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 shutdown_connections 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb7406c820 0x7ffb7406ece0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb88143840 0x7ffb880aa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb88139710 0x7ffb881d04f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 --2- 172.21.15.129:0/2628337485 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb880aef00 0x7ffb881cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 >> 172.21.15.129:0/2628337485 conn(0x7ffb880b03b0 msgr2=0x7ffb880ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 shutdown_connections 2023-12-07T23:46:17.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 -- 172.21.15.129:0/2628337485 wait complete. 2023-12-07T23:46:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.351+0000 7ffb8ec2d640 1 librados: shutdown 2023-12-07T23:46:17.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.3 2023-12-07T23:46:17.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.417+0000 7f677678e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:17.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.417+0000 7f677678e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:17.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.417+0000 7f677678e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.417+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f677013e2f0 con 0x7f6770070210 2023-12-07T23:46:17.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.434+0000 7f9f6d232640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:17.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.434+0000 7f9f6d232640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:17.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.434+0000 7f9f6d232640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.435+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9f6804e840 con 0x7f9f681dbd40 2023-12-07T23:46:17.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.491+0000 7fc160ff9640 1 -- 172.21.15.129:0/4129620774 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc154076ab0 con 0x7fc1640a8970 2023-12-07T23:46:17.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.491+0000 7fc160ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.491+0000 7fc160ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:17.504 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.499+0000 7f6765ffb640 1 -- 172.21.15.129:0/732287447 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f676c0586e0 con 0x7f6770070210 2023-12-07T23:46:17.504 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.499+0000 7f6765ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:17.504 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.499+0000 7f6765ffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.505 INFO:teuthology.orchestra.run.smithi129.stdout:55834574851 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.502+0000 7f677678e640 10 librados: watch_flush enter 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.502+0000 7f677678e640 10 librados: watch_flush exit 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.502+0000 7f677678e640 20 client.4342.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.502+0000 7f677678e640 10 client.4342.objecter successfully canceled tick 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f675406c870 msgr2=0x7f675406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f675406c870 0x7f675406ed30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67701ca6b0 tx=0x7f676800b040 comp rx=0 tx=0).stop 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 10 monclient: shutdown 2023-12-07T23:46:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6770070210 msgr2=0x7f677013e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6770070210 0x7f677013e500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f676c00b6f0 tx=0x7f676c009e00 comp rx=0 tx=0).stop 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 shutdown_connections 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f675406c870 0x7f675406ed30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f677013d760 0x7f677013dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67700aee70 0x7f67701da5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 --2- 172.21.15.129:0/732287447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6770070210 0x7f677013e500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 >> 172.21.15.129:0/732287447 conn(0x7f67700b0630 msgr2=0x7f677006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 shutdown_connections 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 -- 172.21.15.129:0/732287447 wait complete. 2023-12-07T23:46:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.503+0000 7f677678e640 1 librados: shutdown 2023-12-07T23:46:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.509+0000 7f9f4f7fe640 1 -- 172.21.15.129:0/3367175026 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9f5c058300 con 0x7f9f681dbd40 2023-12-07T23:46:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.509+0000 7f9f4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.509+0000 7f9f4f7fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.512 INFO:teuthology.orchestra.run.smithi129.stdout:68719476741 2023-12-07T23:46:17.515 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 10 librados: watch_flush enter 2023-12-07T23:46:17.515 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 10 librados: watch_flush exit 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 20 client.4345.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 10 client.4345.objecter successfully canceled tick 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9f5406c820 msgr2=0x7f9f5406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9f5406c820 0x7f9f5406ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f600012e0 tx=0x7f9f60007d30 comp rx=0 tx=0).stop 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 10 monclient: shutdown 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f681dbd40 msgr2=0x7f9f6813a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f681dbd40 0x7f9f6813a3f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00d490 tx=0x7f9f5c00d950 comp rx=0 tx=0).stop 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 shutdown_connections 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9f5406c820 0x7f9f5406ece0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f681dbd40 0x7f9f6813a3f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f680a8b80 0x7f9f681db800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 --2- 172.21.15.129:0/3367175026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f68072430 0x7f9f6806efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 >> 172.21.15.129:0/3367175026 conn(0x7f9f6806aaf0 msgr2=0x7f9f68140610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 shutdown_connections 2023-12-07T23:46:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 -- 172.21.15.129:0/3367175026 wait complete. 2023-12-07T23:46:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.512+0000 7f9f6d232640 1 librados: shutdown 2023-12-07T23:46:17.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.4 2023-12-07T23:46:17.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.2 2023-12-07T23:46:17.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.535+0000 7f5afe3f0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T23:46:17.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.535+0000 7f5afe3f0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T23:46:17.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.535+0000 7f5afe3f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:17.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.535+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5af80adbf0 con 0x7f5af80a8970 2023-12-07T23:46:17.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.541+0000 7f5af4ff9640 1 -- 172.21.15.129:0/2689208435 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ae8044090 con 0x7f5af80a8970 2023-12-07T23:46:17.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.541+0000 7f5af4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T23:46:17.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.541+0000 7f5af4ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.545 INFO:teuthology.orchestra.run.smithi129.stdout:55834574853 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 10 librados: watch_flush enter 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 10 librados: watch_flush exit 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 20 client.4277.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 10 client.4277.objecter successfully canceled tick 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5ae406c870 msgr2=0x7f5ae406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5ae406c870 0x7f5ae406ed30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5af813b890 tx=0x7f5aec006c90 comp rx=0 tx=0).stop 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 10 monclient: shutdown 2023-12-07T23:46:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 msgr2=0x7f5af813a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813a760 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009870 tx=0x7f5ae80046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 shutdown_connections 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5ae406c870 0x7f5ae406ed30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5af81421a0 0x7f5af80b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5af80a9bd0 0x7f5af80ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 --2- 172.21.15.129:0/2689208435 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5af80a8970 0x7f5af813a760 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.543+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 >> 172.21.15.129:0/2689208435 conn(0x7f5af8066dd0 msgr2=0x7f5af813b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.544+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 shutdown_connections 2023-12-07T23:46:17.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.544+0000 7f5afe3f0640 1 -- 172.21.15.129:0/2689208435 wait complete. 2023-12-07T23:46:17.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.544+0000 7f5afe3f0640 1 librados: shutdown 2023-12-07T23:46:17.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2023-12-07T23:46:17.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.580+0000 7f379df16640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T23:46:17.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.580+0000 7f379df16640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T23:46:17.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.580+0000 7f379df16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:17.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.580+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3790035260 con 0x7f3790033550 2023-12-07T23:46:17.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.618+0000 7fda76ffd640 1 -- 172.21.15.129:0/1278061754 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fda6804e450 con 0x7fda800b3390 2023-12-07T23:46:17.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.618+0000 7fda76ffd640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fda6006c820 0x7fda6006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:17.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.618+0000 7fda7e59c640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fda6006c820 0x7fda6006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:17.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.619+0000 7fda7e59c640 10 monclient: get_auth_request con 0x7fda6006c820 auth_method 0 2023-12-07T23:46:17.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.619+0000 7fda76ffd640 10 client.4335.objecter ms_handle_connect 0x7fda6006c820 2023-12-07T23:46:17.622 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.619+0000 7fda7e59c640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fda6006c820 0x7fda6006ece0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda70001560 tx=0x7fda70007990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:17.635 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 1 -- 172.21.15.129:0/1278061754 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fda68032e30 con 0x7fda800b3390 2023-12-07T23:46:17.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 10 client.4335.objecter ms_dispatch 0x7fda80000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:17.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 3 client.4335.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:17.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 3 client.4335.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:17.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 20 client.4335.objecter dump_active .. 0 homeless 2023-12-07T23:46:17.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 1 -- 172.21.15.129:0/1278061754 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda68076a70 con 0x7fda800b3390 2023-12-07T23:46:17.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:17.636 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.632+0000 7fda76ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:17.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.641+0000 7f3783fff640 1 -- 172.21.15.129:0/2040473104 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3790035260 con 0x7f3790033550 2023-12-07T23:46:17.645 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.641+0000 7f3783fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T23:46:17.645 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.641+0000 7f3783fff640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.645 INFO:teuthology.orchestra.run.smithi129.stdout:60129542147 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 10 librados: watch_flush enter 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 10 librados: watch_flush exit 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 20 client.4332.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 10 client.4332.objecter successfully canceled tick 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f377c06c870 msgr2=0x7f377c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f377c06c870 0x7f377c06ed30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3798068540 tx=0x7f3798068400 comp rx=0 tx=0).stop 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 10 monclient: shutdown 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3790033550 msgr2=0x7f37901655a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3790033550 0x7f37901655a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37900344e0 tx=0x7f37840046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 shutdown_connections 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f377c06c870 0x7f377c06ed30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37900d01b0 0x7f3790165ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3790033550 0x7f37901655a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 --2- 172.21.15.129:0/2040473104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37900071c0 0x7f37900c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 >> 172.21.15.129:0/2040473104 conn(0x7f3790006190 msgr2=0x7f379003b210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 shutdown_connections 2023-12-07T23:46:17.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 -- 172.21.15.129:0/2040473104 wait complete. 2023-12-07T23:46:17.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.644+0000 7f379df16640 1 librados: shutdown 2023-12-07T23:46:17.659 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.1 2023-12-07T23:46:17.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.680+0000 7fc16a64b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T23:46:17.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.680+0000 7fc16a64b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T23:46:17.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.680+0000 7fc16a64b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:17.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.680+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc1641da3f0 con 0x7fc1640a8970 2023-12-07T23:46:17.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.701+0000 7fc160ff9640 1 -- 172.21.15.129:0/4129620774 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc1641da3f0 con 0x7fc1640a8970 2023-12-07T23:46:17.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.701+0000 7fc160ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T23:46:17.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.701+0000 7fc160ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.705 INFO:teuthology.orchestra.run.smithi129.stdout:55834574851 2023-12-07T23:46:17.707 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.703+0000 7fc16a64b640 10 librados: watch_flush enter 2023-12-07T23:46:17.707 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 10 librados: watch_flush exit 2023-12-07T23:46:17.707 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 20 client.4360.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.707 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 10 client.4360.objecter successfully canceled tick 2023-12-07T23:46:17.707 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc15006c870 msgr2=0x7fc15006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc15006c870 0x7fc15006ed30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1640abb00 tx=0x7fc158006c90 comp rx=0 tx=0).stop 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 10 monclient: shutdown 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 msgr2=0x7fc1641daab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc1641daab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc1540046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 shutdown_connections 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc15006c870 0x7fc15006ed30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1641421a0 0x7fc164133e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1640a9bd0 0x7fc164133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 --2- 172.21.15.129:0/4129620774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1640a8970 0x7fc1641daab0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 >> 172.21.15.129:0/4129620774 conn(0x7fc164066dd0 msgr2=0x7fc16413be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 shutdown_connections 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 -- 172.21.15.129:0/4129620774 wait complete. 2023-12-07T23:46:17.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.704+0000 7fc16a64b640 1 librados: shutdown 2023-12-07T23:46:17.720 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2023-12-07T23:46:17.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.822+0000 7fda84f00640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T23:46:17.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.822+0000 7fda84f00640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T23:46:17.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.822+0000 7fda84f00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:17.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.822+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fda8006f850 con 0x7fda800b3390 2023-12-07T23:46:17.828 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.825+0000 7fda76ffd640 1 -- 172.21.15.129:0/1278061754 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fda8006f850 con 0x7fda800b3390 2023-12-07T23:46:17.828 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.825+0000 7fda76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T23:46:17.828 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.825+0000 7fda76ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:68719476741 2023-12-07T23:46:17.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.827+0000 7fda84f00640 10 librados: watch_flush enter 2023-12-07T23:46:17.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.827+0000 7fda84f00640 10 librados: watch_flush exit 2023-12-07T23:46:17.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.827+0000 7fda84f00640 20 client.4335.objecter shutdown clearing up homeless session... 2023-12-07T23:46:17.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.827+0000 7fda84f00640 10 client.4335.objecter successfully canceled tick 2023-12-07T23:46:17.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fda6006c820 msgr2=0x7fda6006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fda6006c820 0x7fda6006ece0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda70001560 tx=0x7fda70007990 comp rx=0 tx=0).stop 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 10 monclient: shutdown 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda800b3390 msgr2=0x7fda80140670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda800b3390 0x7fda80140670 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fda68033000 tx=0x7fda680046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 shutdown_connections 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fda6006c820 0x7fda6006ece0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda80143be0 0x7fda80146330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda800b3390 0x7fda80140670 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 --2- 172.21.15.129:0/1278061754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda8006e970 0x7fda80139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 >> 172.21.15.129:0/1278061754 conn(0x7fda800b7710 msgr2=0x7fda800b7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 shutdown_connections 2023-12-07T23:46:17.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 -- 172.21.15.129:0/1278061754 wait complete. 2023-12-07T23:46:17.833 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:17.828+0000 7fda84f00640 1 librados: shutdown 2023-12-07T23:46:17.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.0 2023-12-07T23:46:18.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-07T23:46:18.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-07T23:46:18.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.396+0000 7f771644b640 10 monclient: get_monmap_and_config 2023-12-07T23:46:18.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.396+0000 7f771644b640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.396+0000 7f771644b640 10 monclient: monmap: 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:18.397722+0000 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:18.397722+0000 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:18.400 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.397+0000 7f771644b640 1 Processor -- start 2023-12-07T23:46:18.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.397+0000 7f771644b640 1 -- start start 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.397+0000 7f771644b640 10 monclient: init 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.397+0000 7f771644b640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.397+0000 7f771644b640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77100a8950 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77100a8d30 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f771013f820 0x7f77100b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 10 monclient(hunting): picked mon.noname-c con 0x7f771013f820 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7710141e30 con 0x7f77100a8d30 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77100aa890 con 0x7f77100a8950 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7710142cb0 con 0x7f771013f820 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f771644b640 10 monclient(hunting): authenticate will time out at 1378.586201s 2023-12-07T23:46:18.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f77149e8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f771013f820 0x7f77100b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f770f7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f77100a8d30 0x7f771013f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43150/0 (socket says 172.21.15.129:43150) 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.398+0000 7f770f7fe640 1 -- 172.21.15.129:0/1475309837 learned_addr learned my addr 172.21.15.129:0/1475309837 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.399+0000 7f77149e8640 10 monclient(hunting): get_auth_request con 0x7f771013f820 auth_method 0 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.399+0000 7f77149e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.399+0000 7f77149e8640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.399+0000 7f77149e8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.399+0000 7f77149e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.399+0000 7f77149e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.399+0000 7f77149e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f770effd640 1 -- 172.21.15.129:0/1475309837 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234068457 0 0) 0x7f7710141e30 con 0x7f77100a8d30 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f770effd640 10 monclient(hunting): _negotiate 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f770effd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f770effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f770effd640 10 monclient(hunting): my global_id is 4338 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f770effd640 1 -- 172.21.15.129:0/1475309837 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f8003520 con 0x7f77100a8d30 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2023-12-07T23:46:18.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 1 -- 172.21.15.129:0/1475309837 >> v1:172.21.15.184:6789/0 conn(0x7f77100a8950 legacy=0x7f77101425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 1 -- 172.21.15.129:0/1475309837 >> v1:172.21.15.129:6789/0 conn(0x7f77100a8d30 legacy=0x7f771013f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 1 monclient: found mon.noname-c 2023-12-07T23:46:18.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.401+0000 7f77149e8640 1 -- 172.21.15.129:0/1475309837 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100aab00 con 0x7f771013f820 2023-12-07T23:46:18.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.405+0000 7f77149e8640 1 --2- 172.21.15.129:0/1475309837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f771013f820 0x7f77100b5610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7708009d90 tx=0x7f770802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcbf7dce8cbdd7fd server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 1 -- 172.21.15.129:0/1475309837 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770802cd00 con 0x7f771013f820 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: dump: 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 1 -- 172.21.15.129:0/1475309837 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770802ce60 con 0x7f771013f820 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 5 monclient: authenticate success, global_id 4363 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 1 -- 172.21.15.129:0/1475309837 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708034660 con 0x7f771013f820 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.414 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 4 set_mon_vals no callback set 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f770effd640 10 monclient: dump: 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 10 monclient: shutdown 2023-12-07T23:46:18.415 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 1 -- 172.21.15.129:0/1475309837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f771013f820 msgr2=0x7f77100b5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 1 --2- 172.21.15.129:0/1475309837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f771013f820 0x7f77100b5610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7708009d90 tx=0x7f770802bf50 comp rx=0 tx=0).stop 2023-12-07T23:46:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.409+0000 7f771644b640 1 -- 172.21.15.129:0/1475309837 shutdown_connections 2023-12-07T23:46:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 1 --2- 172.21.15.129:0/1475309837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f771013f820 0x7f77100b5610 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 1 -- 172.21.15.129:0/1475309837 >> 172.21.15.129:0/1475309837 conn(0x7f7710066cd0 msgr2=0x7f77100b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 1 -- 172.21.15.129:0/1475309837 shutdown_connections 2023-12-07T23:46:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 1 -- 172.21.15.129:0/1475309837 wait complete. 2023-12-07T23:46:18.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 10 monclient: monmap: 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 1 librados: starting msgr at 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.410+0000 7f771644b640 1 librados: starting objecter 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 Processor -- start 2023-12-07T23:46:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 -- start start 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 librados: setting wanted keys 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 librados: calling monclient init 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient: init 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77100a8950 0x7f7710141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): picked mon.noname-a con 0x7f77100a8950 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77100a8d30 0x7f7710138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): picked mon.noname-b con 0x7f77100a8d30 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f771013f820 0x7f7710145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): picked mon.noname-c con 0x7f771013f820 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77101325f0 con 0x7f77100a8950 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77101375f0 con 0x7f77100a8d30 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7710138740 con 0x7f771013f820 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.411+0000 7f771644b640 10 monclient(hunting): authenticate will time out at 1378.599840s 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77100a8d30 0x7f7710138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f77149e8640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f771013f820 0x7f7710145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77100a8950 0x7f7710141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77100a8d30 0x7f7710138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40742/0 (socket says 172.21.15.129:40742) 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770f7fe640 1 -- 172.21.15.129:0/456718907 learned_addr learned my addr 172.21.15.129:0/456718907 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770f7fe640 10 monclient(hunting): get_auth_request con 0x7f77100a8d30 auth_method 0 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f770f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f77149e8640 10 monclient(hunting): get_auth_request con 0x7f771013f820 auth_method 0 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f77149e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f77149e8640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.412+0000 7f77149e8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770ffff640 10 monclient(hunting): get_auth_request con 0x7f77100a8950 auth_method 0 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770ffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f77149e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f77149e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f77149e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.413+0000 7f770ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 1 -- 172.21.15.129:0/456718907 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f771013f820 msgr2=0x7f7710145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f771013f820 0x7f7710145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 1 -- 172.21.15.129:0/456718907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77100a8950 msgr2=0x7f7710141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77100a8950 0x7f7710141960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 1 monclient: found mon.noname-b 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.415+0000 7f770f7fe640 1 -- 172.21.15.129:0/456718907 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7708009a70 con 0x7f77100a8d30 2023-12-07T23:46:18.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.417+0000 7f770f7fe640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77100a8d30 0x7f7710138200 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f770000c430 tx=0x7f770000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.417+0000 7f770cff9640 10 client.?.objecter ms_handle_connect 0x7f77100a8d30 2023-12-07T23:46:18.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.417+0000 7f770cff9640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 1 -- 172.21.15.129:0/456718907 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700005ce0 con 0x7f77100a8d30 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: dump: 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 1 -- 172.21.15.129:0/456718907 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7700005e40 con 0x7f77100a8d30 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 1 -- 172.21.15.129:0/456718907 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77000072f0 con 0x7f77100a8d30 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770cff9640 10 monclient: dump: 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.427 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 5 monclient: authenticate success, global_id 4366 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 10 monclient: _renew_subs 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f770dffb640 4 set_mon_vals no callback set 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7710137b00 con 0x7f77100a8d30 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 10 monclient: _renew_subs 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7710145f80 con 0x7f77100a8d30 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.421+0000 7f771644b640 1 librados: init done 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.422+0000 7f771644b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.422+0000 7f771644b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.422+0000 7f771644b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.429 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.422+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77100b72b0 con 0x7f77100a8d30 2023-12-07T23:46:18.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.439+0000 7f770cff9640 1 -- 172.21.15.129:0/456718907 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7700007450 con 0x7f77100a8d30 2023-12-07T23:46:18.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.439+0000 7f770cff9640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f76fc06c820 0x7f76fc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.440+0000 7f770ffff640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f76fc06c820 0x7f76fc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.440+0000 7f770ffff640 10 monclient: get_auth_request con 0x7f76fc06c820 auth_method 0 2023-12-07T23:46:18.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.440+0000 7f770cff9640 10 client.4366.objecter ms_handle_connect 0x7f76fc06c820 2023-12-07T23:46:18.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.440+0000 7f770ffff640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f76fc06c820 0x7f76fc06ece0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7704000f60 tx=0x7f7704009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.442+0000 7f770cff9640 1 -- 172.21.15.129:0/456718907 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7700057eb0 con 0x7f77100a8d30 2023-12-07T23:46:18.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.442+0000 7f770cff9640 10 client.4366.objecter ms_dispatch 0x7f7710000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:18.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.442+0000 7f770cff9640 3 client.4366.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:18.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.442+0000 7f770cff9640 3 client.4366.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:18.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.442+0000 7f770cff9640 20 client.4366.objecter dump_active .. 0 homeless 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.473+0000 7fb36af3e640 10 monclient: get_monmap_and_config 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.473+0000 7fb36af3e640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.473+0000 7fb36af3e640 10 monclient: monmap: 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:18.474204+0000 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:18.474204+0000 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.476 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:18.477 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:18.477 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:18.477 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:18.477 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:18.477 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.478 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.473+0000 7fb36af3e640 1 Processor -- start 2023-12-07T23:46:18.478 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.473+0000 7fb36af3e640 1 -- start start 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.473+0000 7fb36af3e640 10 monclient: init 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb3640a8970 0x7fb36413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3640a8970 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3640a9bd0 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient(hunting): picked mon.noname-b con 0x7fb36413f3b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3640aa950 con 0x7fb3640a9bd0 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb364142df0 con 0x7fb3640a8970 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640a90e0 con 0x7fb36413f3b0 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.474+0000 7fb36af3e640 10 monclient(hunting): authenticate will time out at 1378.662589s 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.475+0000 7fb3694db640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.475+0000 7fb3694db640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55680/0 (socket says 172.21.15.129:55680) 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.475+0000 7fb3694db640 1 -- 172.21.15.129:0/2735412997 learned_addr learned my addr 172.21.15.129:0/2735412997 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.475+0000 7fb3694db640 10 monclient(hunting): get_auth_request con 0x7fb36413f3b0 auth_method 0 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.475+0000 7fb3694db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.475+0000 7fb3694db640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.475+0000 7fb3694db640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.476+0000 7fb3694db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.476+0000 7fb3694db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.476+0000 7fb3694db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2023-12-07T23:46:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 1 -- 172.21.15.129:0/2735412997 >> v2:172.21.15.184:3300/0 conn(0x7fb3640a8970 msgr2=0x7fb36413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 1 --2- 172.21.15.129:0/2735412997 >> v2:172.21.15.184:3300/0 conn(0x7fb3640a8970 0x7fb36413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 1 -- 172.21.15.129:0/2735412997 >> v1:172.21.15.184:6789/0 conn(0x7fb3640a9bd0 legacy=0x7fb3641421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 1 monclient: found mon.noname-b 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.477+0000 7fb3694db640 1 -- 172.21.15.129:0/2735412997 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36413f800 con 0x7fb36413f3b0 2023-12-07T23:46:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.479+0000 7fb3694db640 1 --2- 172.21.15.129:0/2735412997 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641428b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009e50 tx=0x7fb35c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=594e58cf68193caf server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 1 -- 172.21.15.129:0/2735412997 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c02c950 con 0x7fb36413f3b0 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: dump: 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.486 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb36af3e640 5 monclient: authenticate success, global_id 4283 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb36af3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 1 -- 172.21.15.129:0/2735412997 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c02cab0 con 0x7fb36413f3b0 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb36af3e640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb36af3e640 4 set_mon_vals no callback set 2023-12-07T23:46:18.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 1 -- 172.21.15.129:0/2735412997 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c02ce40 con 0x7fb36413f3b0 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.482+0000 7fb3637fe640 10 monclient: dump: 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.488 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 10 monclient: shutdown 2023-12-07T23:46:18.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:18.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 -- 172.21.15.129:0/2735412997 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 msgr2=0x7fb3641428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.492 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 --2- 172.21.15.129:0/2735412997 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641428b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009e50 tx=0x7fb35c02bdb0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 -- 172.21.15.129:0/2735412997 shutdown_connections 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 --2- 172.21.15.129:0/2735412997 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641428b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 --2- 172.21.15.129:0/2735412997 >> v2:172.21.15.184:3300/0 conn(0x7fb3640a8970 0x7fb36413ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 -- 172.21.15.129:0/2735412997 >> 172.21.15.129:0/2735412997 conn(0x7fb364066dd0 msgr2=0x7fb3640b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 -- 172.21.15.129:0/2735412997 shutdown_connections 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.483+0000 7fb36af3e640 1 -- 172.21.15.129:0/2735412997 wait complete. 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 10 monclient: monmap: 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:18.493 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 1 librados: starting msgr at 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 1 librados: starting objecter 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 1 Processor -- start 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 1 -- start start 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 1 librados: setting wanted keys 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 1 librados: calling monclient init 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.484+0000 7fb36af3e640 10 monclient: init 2023-12-07T23:46:18.494 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3640a8970 0x7fb364137d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3640a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640a9bd0 0x7fb3641d7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3640a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641d7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): picked mon.noname-c con 0x7fb36413f3b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb364139180 con 0x7fb3640a9bd0 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3641431f0 con 0x7fb3640a8970 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb364139380 con 0x7fb36413f3b0 2023-12-07T23:46:18.495 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb36af3e640 10 monclient(hunting): authenticate will time out at 1378.673338s 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb3694db640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641d7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.485+0000 7fb363fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640a9bd0 0x7fb3641d7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641d7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55694/0 (socket says 172.21.15.129:55694) 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 1 -- 172.21.15.129:0/909364438 learned_addr learned my addr 172.21.15.129:0/909364438 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 10 monclient(hunting): get_auth_request con 0x7fb36413f3b0 auth_method 0 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb363fff640 10 monclient(hunting): get_auth_request con 0x7fb3640a9bd0 auth_method 0 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb363fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb363fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb363fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.496 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb3694db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb363fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb363fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.486+0000 7fb363fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 1 -- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3640a8970 msgr2=0x7fb364137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3640a8970 0x7fb364137d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 1 -- 172.21.15.129:0/909364438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640a9bd0 msgr2=0x7fb3641d7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640a9bd0 0x7fb3641d7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.497 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 1 monclient: found mon.noname-c 2023-12-07T23:46:18.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.498 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.489+0000 7fb3694db640 1 -- 172.21.15.129:0/909364438 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c009b30 con 0x7fb36413f3b0 2023-12-07T23:46:18.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.491+0000 7fb3694db640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641d7950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009e50 tx=0x7fb35c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.499 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.491+0000 7fb3617fa640 10 client.?.objecter ms_handle_connect 0x7fb36413f3b0 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.491+0000 7fb3617fa640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 1 -- 172.21.15.129:0/909364438 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c009510 con 0x7fb36413f3b0 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: dump: 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.500 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 1 -- 172.21.15.129:0/909364438 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c03e450 con 0x7fb36413f3b0 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 5 monclient: authenticate success, global_id 4286 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 10 monclient: _renew_subs 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb364139660 con 0x7fb36413f3b0 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 1 -- 172.21.15.129:0/909364438 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c03e7e0 con 0x7fb36413f3b0 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb3617fa640 10 monclient: dump: 2023-12-07T23:46:18.501 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb362ffd640 4 set_mon_vals no callback set 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 10 monclient: _renew_subs 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.502 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3641d7e90 con 0x7fb36413f3b0 2023-12-07T23:46:18.503 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.495+0000 7fb36af3e640 1 librados: init done 2023-12-07T23:46:18.503 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.496+0000 7fb36af3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.503 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.496+0000 7fb36af3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.503 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.496+0000 7fb36af3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.503 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.496+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36404e840 con 0x7fb36413f3b0 2023-12-07T23:46:18.518 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.514+0000 7fb3617fa640 1 -- 172.21.15.129:0/909364438 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb35c03e940 con 0x7fb36413f3b0 2023-12-07T23:46:18.518 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.515+0000 7fb3617fa640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb35006c8a0 0x7fb35006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.518 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.515+0000 7fb368cda640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb35006c8a0 0x7fb35006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.518 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.515+0000 7fb368cda640 10 monclient: get_auth_request con 0x7fb35006c8a0 auth_method 0 2023-12-07T23:46:18.519 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.515+0000 7fb368cda640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb35006c8a0 0x7fb35006ed60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb354009710 tx=0x7fb354006c90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.519 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.515+0000 7fb3617fa640 10 client.4286.objecter ms_handle_connect 0x7fb35006c8a0 2023-12-07T23:46:18.522 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.519+0000 7fb3617fa640 1 -- 172.21.15.129:0/909364438 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb35c07bf50 con 0x7fb36413f3b0 2023-12-07T23:46:18.523 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.519+0000 7fb3617fa640 10 client.4286.objecter ms_dispatch 0x7fb364000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:18.523 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.519+0000 7fb3617fa640 3 client.4286.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:18.523 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.519+0000 7fb3617fa640 3 client.4286.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:18.523 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.519+0000 7fb3617fa640 20 client.4286.objecter dump_active .. 0 homeless 2023-12-07T23:46:18.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-07T23:46:18.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-07T23:46:18.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-07T23:46:18.626 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.623+0000 7f82beeb4640 10 monclient: get_monmap_and_config 2023-12-07T23:46:18.626 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.623+0000 7f82beeb4640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.626 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.623+0000 7f82beeb4640 10 monclient: monmap: 2023-12-07T23:46:18.626 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.626 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.626 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:18.624357+0000 2023-12-07T23:46:18.626 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:18.624357+0000 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:18.627 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.628 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.623+0000 7f82beeb4640 1 Processor -- start 2023-12-07T23:46:18.628 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 1 -- start start 2023-12-07T23:46:18.628 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient: init 2023-12-07T23:46:18.628 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.628 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 0x7f82b80a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient(hunting): picked mon.noname-b con 0x7f82b80a8950 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f82b80b5610 0x7f82b81425a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient(hunting): picked mon.noname-a con 0x7f82b80b5610 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82b8142ae0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b8141e30 con 0x7f82b8142ae0 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b80b5a30 con 0x7f82b80b5610 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80aa890 con 0x7f82b80a8950 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.624+0000 7f82beeb4640 10 monclient(hunting): authenticate will time out at 1378.812646s 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.625+0000 7f82b7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f82b80b5610 0x7f82b81425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.625+0000 7f82b7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f82b80b5610 0x7f82b81425a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43014/0 (socket says 172.21.15.129:43014) 2023-12-07T23:46:18.629 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.625+0000 7f82b7fff640 1 -- 172.21.15.129:0/2648055524 learned_addr learned my addr 172.21.15.129:0/2648055524 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.625+0000 7f82b7fff640 10 monclient(hunting): get_auth_request con 0x7f82b80b5610 auth_method 0 2023-12-07T23:46:18.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.625+0000 7f82b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.625+0000 7f82b7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.625+0000 7f82b7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.626+0000 7f82b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.626+0000 7f82b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.630 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.626+0000 7f82b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2023-12-07T23:46:18.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 1 -- 172.21.15.129:0/2648055524 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 msgr2=0x7f82b80a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 1 --2- 172.21.15.129:0/2648055524 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 0x7f82b80a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 1 -- 172.21.15.129:0/2648055524 >> v1:172.21.15.129:6789/0 conn(0x7f82b8142ae0 legacy=0x7f82b813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.632 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 1 monclient: found mon.noname-a 2023-12-07T23:46:18.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:18.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.629+0000 7f82b7fff640 1 -- 172.21.15.129:0/2648055524 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b80aab00 con 0x7f82b80b5610 2023-12-07T23:46:18.637 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.635+0000 7f82b7fff640 1 --2- 172.21.15.129:0/2648055524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81425a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009870 tx=0x7f82a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f332ce594ebb6b server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 1 -- 172.21.15.129:0/2648055524 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4031640 con 0x7f82b80b5610 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: dump: 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 1 -- 172.21.15.129:0/2648055524 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a40317a0 con 0x7f82b80b5610 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82beeb4640 5 monclient: authenticate success, global_id 4347 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82beeb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82beeb4640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82beeb4640 4 set_mon_vals no callback set 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 1 -- 172.21.15.129:0/2648055524 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4031b30 con 0x7f82b80b5610 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.640+0000 7f82b77fe640 10 monclient: dump: 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.645 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 10 monclient: shutdown 2023-12-07T23:46:18.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:18.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 -- 172.21.15.129:0/2648055524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 msgr2=0x7f82b81425a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 --2- 172.21.15.129:0/2648055524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81425a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009870 tx=0x7f82a402b8f0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 -- 172.21.15.129:0/2648055524 shutdown_connections 2023-12-07T23:46:18.648 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 --2- 172.21.15.129:0/2648055524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81425a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 --2- 172.21.15.129:0/2648055524 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 0x7f82b80a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 -- 172.21.15.129:0/2648055524 >> 172.21.15.129:0/2648055524 conn(0x7f82b8066cd0 msgr2=0x7f82b80b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 -- 172.21.15.129:0/2648055524 shutdown_connections 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.641+0000 7f82beeb4640 1 -- 172.21.15.129:0/2648055524 wait complete. 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.642+0000 7f82beeb4640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.642+0000 7f82beeb4640 10 monclient: monmap: 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:18.649 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.642+0000 7f82beeb4640 1 librados: starting msgr at 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.642+0000 7f82beeb4640 1 librados: starting objecter 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.642+0000 7f82beeb4640 1 Processor -- start 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.642+0000 7f82beeb4640 1 -- start start 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.643+0000 7f82beeb4640 1 librados: setting wanted keys 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.643+0000 7f82beeb4640 1 librados: calling monclient init 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.643+0000 7f82beeb4640 10 monclient: init 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 0x7f82b813b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): picked mon.noname-c con 0x7f82b80a8950 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81dab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): picked mon.noname-a con 0x7f82b80b5610 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:18.650 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82b81db0c0 0x7f82b812f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): picked mon.noname-b con 0x7f82b81db0c0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80b3600 con 0x7f82b80b5610 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b813ba10 con 0x7f82b81db0c0 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b81db600 con 0x7f82b80a8950 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82beeb4640 10 monclient(hunting): authenticate will time out at 1378.832164s 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.644+0000 7f82b7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81dab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81dab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43016/0 (socket says 172.21.15.129:43016) 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 1 -- 172.21.15.129:0/2342258278 learned_addr learned my addr 172.21.15.129:0/2342258278 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 10 monclient(hunting): get_auth_request con 0x7f82b80b5610 auth_method 0 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.651 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.645+0000 7f82b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2023-12-07T23:46:18.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 1 -- 172.21.15.129:0/2342258278 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 msgr2=0x7f82b813b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 0x7f82b813b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 1 -- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82b81db0c0 msgr2=0x7f82b812f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82b81db0c0 0x7f82b812f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 1 monclient: found mon.noname-a 2023-12-07T23:46:18.654 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.654 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:18.654 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.650+0000 7f82b7fff640 1 -- 172.21.15.129:0/2342258278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a40094d0 con 0x7f82b80b5610 2023-12-07T23:46:18.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.657+0000 7f82b7fff640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81dab80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009870 tx=0x7f82a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.657+0000 7f82b57fa640 10 client.?.objecter ms_handle_connect 0x7f82b80b5610 2023-12-07T23:46:18.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.657+0000 7f82b57fa640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:18.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-07T23:46:18.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 1 -- 172.21.15.129:0/2342258278 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a403ac60 con 0x7f82b80b5610 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 10 monclient: dump: 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.666 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 1 -- 172.21.15.129:0/2342258278 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a403adc0 con 0x7f82b80b5610 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82beeb4640 5 monclient: authenticate success, global_id 4350 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82beeb4640 10 monclient: _renew_subs 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82beeb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82beeb4640 1 -- 172.21.15.129:0/2342258278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b812fe50 con 0x7f82b80b5610 2023-12-07T23:46:18.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b57fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82b6ffd640 4 set_mon_vals no callback set 2023-12-07T23:46:18.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.662+0000 7f82beeb4640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:18.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.663+0000 7f82b57fa640 1 -- 172.21.15.129:0/2342258278 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4044630 con 0x7f82b80b5610 2023-12-07T23:46:18.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.663+0000 7f82beeb4640 10 monclient: _renew_subs 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.663+0000 7f82beeb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.663+0000 7f82beeb4640 1 -- 172.21.15.129:0/2342258278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b8130310 con 0x7f82b80b5610 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.663+0000 7f82beeb4640 1 librados: init done 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.663+0000 7f82b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.663+0000 7f82b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.665+0000 7f82b57fa640 10 monclient: dump: 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.670 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.665+0000 7f82a2ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.665+0000 7f82a2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.665+0000 7f82a2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:18.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.665+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8290005080 con 0x7f82b80b5610 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.669+0000 7f660021f640 10 monclient: get_monmap_and_config 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.669+0000 7f660021f640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.669+0000 7f660021f640 10 monclient: monmap: 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:18.670747+0000 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:18.670747+0000 2023-12-07T23:46:18.672 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.673 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.673 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:18.673 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:18.673 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:18.673 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:18.673 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:18.673 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.670+0000 7f660021f640 1 Processor -- start 2023-12-07T23:46:18.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.670+0000 7f660021f640 1 -- start start 2023-12-07T23:46:18.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.670+0000 7f660021f640 10 monclient: init 2023-12-07T23:46:18.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65f80a8970 0x7f65f813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient(hunting): picked mon.noname-a con 0x7f65f80a8970 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65f80a9bd0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f65f813f3b0 0x7f65f81428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient(hunting): picked mon.noname-c con 0x7f65f813f3b0 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f80aa950 con 0x7f65f80a9bd0 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f8142df0 con 0x7f65f80a8970 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f80a90e0 con 0x7f65f813f3b0 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.671+0000 7f660021f640 10 monclient(hunting): authenticate will time out at 1378.859269s 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.672+0000 7f3483e94640 10 monclient: get_monmap_and_config 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.672+0000 7f3483e94640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.672+0000 7f3483e94640 10 monclient: monmap: 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:18.673389+0000 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:18.673389+0000 2023-12-07T23:46:18.676 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.673+0000 7f3483e94640 1 Processor -- start 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.673+0000 7f3483e94640 1 -- start start 2023-12-07T23:46:18.677 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.673+0000 7f3483e94640 10 monclient: init 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f347c06a730 0x7f347c0afd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient(hunting): picked mon.noname-c con 0x7f347c06a730 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f347c06fd90 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 0x7f347c150a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient(hunting): picked mon.noname-b con 0x7f347c146de0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c071050 con 0x7f347c06fd90 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c143df0 con 0x7f347c06a730 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c146280 con 0x7f347c146de0 2023-12-07T23:46:18.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f3483e94640 10 monclient(hunting): authenticate will time out at 1378.862716s 2023-12-07T23:46:18.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f770cff9640 1 -- 172.21.15.129:0/456718907 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77000541f0 con 0x7f77100a8d30 2023-12-07T23:46:18.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f770cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.674+0000 7f770cff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:18.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.678+0000 7f3481c30640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f347c06a730 0x7f347c0afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f347c06a730 0x7f347c0afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40804/0 (socket says 172.21.15.129:40804) 2023-12-07T23:46:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 1 -- 172.21.15.129:0/3518763121 learned_addr learned my addr 172.21.15.129:0/3518763121 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 10 monclient(hunting): get_auth_request con 0x7f347c06a730 auth_method 0 2023-12-07T23:46:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.679+0000 7f3481c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 1 -- 172.21.15.129:0/3518763121 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 msgr2=0x7f347c150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 1 --2- 172.21.15.129:0/3518763121 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 0x7f347c150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 1 -- 172.21.15.129:0/3518763121 >> v1:172.21.15.129:6789/0 conn(0x7f347c06fd90 legacy=0x7f347c1466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 1 monclient: found mon.noname-c 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.681+0000 7f3481c30640 1 -- 172.21.15.129:0/3518763121 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c071330 con 0x7f347c06a730 2023-12-07T23:46:18.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.684+0000 7f65fe7bc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f65f813f3b0 0x7f65f81428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.685+0000 7f65fe7bc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f65f813f3b0 0x7f65f81428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40788/0 (socket says 172.21.15.129:40788) 2023-12-07T23:46:18.687 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.685+0000 7f65fe7bc640 1 -- 172.21.15.129:0/396169271 learned_addr learned my addr 172.21.15.129:0/396169271 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.685+0000 7f65fe7bc640 10 monclient(hunting): get_auth_request con 0x7f65f813f3b0 auth_method 0 2023-12-07T23:46:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.685+0000 7f65fe7bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.685+0000 7f65fe7bc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.685+0000 7f65fe7bc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.686+0000 7f65fe7bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.686+0000 7f65fe7bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.686+0000 7f65fe7bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.686+0000 7f82b57fa640 1 -- 172.21.15.129:0/2342258278 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f82a402f050 con 0x7f82b80b5610 2023-12-07T23:46:18.690 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.687+0000 7f3481c30640 1 --2- 172.21.15.129:0/3518763121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c0afd80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3478009870 tx=0x7f347802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c76fbe88d4d04a04 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.687+0000 7f82b57fa640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f829c06c870 0x7f829c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.687+0000 7f82bcc50640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f829c06c870 0x7f829c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.687+0000 7f82bcc50640 10 monclient: get_auth_request con 0x7f829c06c870 auth_method 0 2023-12-07T23:46:18.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.688+0000 7f82bcc50640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f829c06c870 0x7f829c06ed30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009670 tx=0x7f82ac006bf0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.691 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.688+0000 7f82b57fa640 10 client.4350.objecter ms_handle_connect 0x7f829c06c870 2023-12-07T23:46:18.692 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2023-12-07T23:46:18.692 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 1 -- 172.21.15.129:0/396169271 >> v2:172.21.15.129:3300/0 conn(0x7f65f80a8970 msgr2=0x7f65f813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.692 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 1 --2- 172.21.15.129:0/396169271 >> v2:172.21.15.129:3300/0 conn(0x7f65f80a8970 0x7f65f813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.692 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 1 -- 172.21.15.129:0/396169271 >> v1:172.21.15.129:6789/0 conn(0x7f65f80a9bd0 legacy=0x7f65f81421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.692 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 1 monclient: found mon.noname-c 2023-12-07T23:46:18.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.693 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.689+0000 7f65fe7bc640 1 -- 172.21.15.129:0/396169271 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f80aabc0 con 0x7f65f813f3b0 2023-12-07T23:46:18.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 1 -- 172.21.15.129:0/3518763121 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34780317c0 con 0x7f347c06a730 2023-12-07T23:46:18.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:18.697 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: dump: 2023-12-07T23:46:18.697 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.698 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 1 -- 172.21.15.129:0/3518763121 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3478031920 con 0x7f347c06a730 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 1 -- 172.21.15.129:0/3518763121 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3478031cb0 con 0x7f347c06a730 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.693+0000 7f3480c2e640 10 monclient: dump: 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.699 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.700 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.700 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.700 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.700 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.700 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.694+0000 7f65fe7bc640 1 --2- 172.21.15.129:0/396169271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f813f3b0 0x7f65f81428b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f65f80b5050 tx=0x7f65f002bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca0d6a78455f8e6e server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.694+0000 7f3483e94640 5 monclient: authenticate success, global_id 4369 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.694+0000 7f3483e94640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.694+0000 7f3483e94640 4 set_mon_vals no callback set 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 10 monclient: shutdown 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 -- 172.21.15.129:0/3518763121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 msgr2=0x7f347c0afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 --2- 172.21.15.129:0/3518763121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c0afd80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3478009870 tx=0x7f347802b860 comp rx=0 tx=0).stop 2023-12-07T23:46:18.702 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 -- 172.21.15.129:0/3518763121 shutdown_connections 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 --2- 172.21.15.129:0/3518763121 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 0x7f347c150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 --2- 172.21.15.129:0/3518763121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c0afd80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 -- 172.21.15.129:0/3518763121 >> 172.21.15.129:0/3518763121 conn(0x7f347c0b4a80 msgr2=0x7f347c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 -- 172.21.15.129:0/3518763121 shutdown_connections 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.695+0000 7f3483e94640 1 -- 172.21.15.129:0/3518763121 wait complete. 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.696+0000 7f3483e94640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.696+0000 7f3483e94640 10 monclient: monmap: 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.703 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.697+0000 7f3483e94640 1 librados: starting msgr at 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.697+0000 7f3483e94640 1 librados: starting objecter 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.697+0000 7f3483e94640 1 Processor -- start 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.697+0000 7f3483e94640 1 -- start start 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.697+0000 7f3483e94640 1 librados: setting wanted keys 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.697+0000 7f3483e94640 1 librados: calling monclient init 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.697+0000 7f3483e94640 10 monclient: init 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.704 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): picked mon.noname-b con 0x7f347c06a730 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06fd90 0x7f347c1d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): picked mon.noname-a con 0x7f347c06fd90 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 0x7f347c1df4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 1 -- 172.21.15.129:0/396169271 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f002c960 con 0x7f65f813f3b0 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: dump: 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.705 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 1 -- 172.21.15.129:0/396169271 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f002cac0 con 0x7f65f813f3b0 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 1 -- 172.21.15.129:0/396169271 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f002ce50 con 0x7f65f813f3b0 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:18.706 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f65fcfb9640 10 monclient: dump: 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.707 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): picked mon.noname-c con 0x7f347c146de0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c1d2e00 con 0x7f347c06fd90 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c1df050 con 0x7f347c06a730 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c1d3030 con 0x7f347c146de0 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.708 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3483e94640 10 monclient(hunting): authenticate will time out at 1378.886579s 2023-12-07T23:46:18.709 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f660021f640 5 monclient: authenticate success, global_id 4372 2023-12-07T23:46:18.709 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f660021f640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:18.709 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f660021f640 4 set_mon_vals no callback set 2023-12-07T23:46:18.709 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.698+0000 7f3481c30640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.709 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c149860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40820/0 (socket says 172.21.15.129:40820) 2023-12-07T23:46:18.709 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 1 -- 172.21.15.129:0/1083168691 learned_addr learned my addr 172.21.15.129:0/1083168691 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 10 monclient(hunting): get_auth_request con 0x7f347c06a730 auth_method 0 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 10 monclient: shutdown 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 1 -- 172.21.15.129:0/396169271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f813f3b0 msgr2=0x7f65f81428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.710 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 1 --2- 172.21.15.129:0/396169271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f813f3b0 0x7f65f81428b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f65f80b5050 tx=0x7f65f002bdc0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 1 -- 172.21.15.129:0/396169271 shutdown_connections 2023-12-07T23:46:18.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 1 --2- 172.21.15.129:0/396169271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f813f3b0 0x7f65f81428b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 1 --2- 172.21.15.129:0/396169271 >> v2:172.21.15.129:3300/0 conn(0x7f65f80a8970 0x7f65f813ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 1 -- 172.21.15.129:0/396169271 >> 172.21.15.129:0/396169271 conn(0x7f65f8066dd0 msgr2=0x7f65f80b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:18.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.711 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f3481c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.699+0000 7f660021f640 1 -- 172.21.15.129:0/396169271 shutdown_connections 2023-12-07T23:46:18.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.701+0000 7f660021f640 1 -- 172.21.15.129:0/396169271 wait complete. 2023-12-07T23:46:18.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.701+0000 7f660021f640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.712 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.701+0000 7f660021f640 10 monclient: monmap: 2023-12-07T23:46:18.712 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.712 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f660021f640 1 librados: starting msgr at 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f660021f640 1 librados: starting objecter 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f660021f640 1 Processor -- start 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f660021f640 1 -- start start 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f660021f640 1 librados: setting wanted keys 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f660021f640 1 librados: calling monclient init 2023-12-07T23:46:18.713 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f660021f640 10 monclient: init 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 1 -- 172.21.15.129:0/1083168691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 msgr2=0x7f347c1df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 0x7f347c1df4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 1 -- 172.21.15.129:0/1083168691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06fd90 msgr2=0x7f347c1d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06fd90 0x7f347c1d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 1 monclient: found mon.noname-b 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.714 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.702+0000 7f3481c30640 1 -- 172.21.15.129:0/1083168691 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34780094d0 con 0x7f347c06a730 2023-12-07T23:46:18.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:18.715 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f80a8970 0x7f65f813a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): picked mon.noname-b con 0x7f65f80a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65f80a9bd0 0x7f65f81da320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): picked mon.noname-a con 0x7f65f80a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65f813f3b0 0x7f65f81d6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): picked mon.noname-c con 0x7f65f813f3b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f81dab60 con 0x7f65f80a9bd0 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f81da860 con 0x7f65f80a8970 2023-12-07T23:46:18.716 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80ae680 con 0x7f65f813f3b0 2023-12-07T23:46:18.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.703+0000 7f660021f640 10 monclient(hunting): authenticate will time out at 1378.891626s 2023-12-07T23:46:18.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.705+0000 7f65fdfbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f80a8970 0x7f65f813a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.705+0000 7f3481c30640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c149860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3478000c00 tx=0x7f34780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.705+0000 7f34727fc640 10 client.?.objecter ms_handle_connect 0x7f347c06a730 2023-12-07T23:46:18.717 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.705+0000 7f34727fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:18.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.706+0000 7f65fdfbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f80a8970 0x7f65f813a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40822/0 (socket says 172.21.15.129:40822) 2023-12-07T23:46:18.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.706+0000 7f65fdfbb640 1 -- 172.21.15.129:0/2365825212 learned_addr learned my addr 172.21.15.129:0/2365825212 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.706+0000 7f65fdfbb640 10 monclient(hunting): get_auth_request con 0x7f65f80a8970 auth_method 0 2023-12-07T23:46:18.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.706+0000 7f65fdfbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.706+0000 7f65fdfbb640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.718 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.706+0000 7f65fdfbb640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.707+0000 7f65fdfbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.707+0000 7f65fdfbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.707+0000 7f65fdfbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 1 -- 172.21.15.129:0/2365825212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65f813f3b0 msgr2=0x7f65f81d6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65f813f3b0 0x7f65f81d6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 1 -- 172.21.15.129:0/2365825212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65f80a9bd0 msgr2=0x7f65f81da320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65f80a9bd0 0x7f65f81da320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 1 monclient: found mon.noname-b 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.719 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.710+0000 7f65fdfbb640 1 -- 172.21.15.129:0/2365825212 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f0009b30 con 0x7f65f80a8970 2023-12-07T23:46:18.720 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.713+0000 7f34727fc640 1 -- 172.21.15.129:0/1083168691 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34780328e0 con 0x7f347c06a730 2023-12-07T23:46:18.720 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.713+0000 7f34727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.720 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f34727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:18.720 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f34727fc640 10 monclient: dump: 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f34727fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f34727fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f34727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 5 monclient: authenticate success, global_id 4375 2023-12-07T23:46:18.721 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 10 monclient: _renew_subs 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c1dfa20 con 0x7f347c06a730 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 10 monclient: _renew_subs 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c13eda0 con 0x7f347c06a730 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f3483e94640 1 librados: init done 2023-12-07T23:46:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f65fdfbb640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f80a8970 0x7f65f813a320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f65f400b4a0 tx=0x7f65f400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f65ee7fc640 10 client.?.objecter ms_handle_connect 0x7f65f80a8970 2023-12-07T23:46:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.714+0000 7f65ee7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f34727fc640 1 -- 172.21.15.129:0/1083168691 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3478032a40 con 0x7f347c06a730 2023-12-07T23:46:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f34727fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f34727fc640 1 -- 172.21.15.129:0/1083168691 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3478032db0 con 0x7f347c06a730 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f34727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f34727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f34727fc640 10 monclient: dump: 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.724 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.725 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f3473fff640 4 set_mon_vals no callback set 2023-12-07T23:46:18.725 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.715+0000 7f3483e94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.725 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.716+0000 7f3483e94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.725 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.716+0000 7f3483e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.725 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.716+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3464005080 con 0x7f347c06a730 2023-12-07T23:46:18.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-07T23:46:18.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.744+0000 7f82b57fa640 1 -- 172.21.15.129:0/2342258278 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f82a407ad70 con 0x7f82b80b5610 2023-12-07T23:46:18.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.744+0000 7f82b57fa640 10 client.4350.objecter ms_dispatch 0x7f82b8000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:18.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.744+0000 7f82b57fa640 3 client.4350.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:18.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.744+0000 7f82b57fa640 3 client.4350.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:18.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.744+0000 7f82b57fa640 20 client.4350.objecter dump_active .. 0 homeless 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.771+0000 7f7079333640 10 monclient: get_monmap_and_config 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.771+0000 7f7079333640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.771+0000 7f7079333640 10 monclient: monmap: 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:18.772300+0000 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:18.772300+0000 2023-12-07T23:46:18.774 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:18.775 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.771+0000 7f7079333640 1 Processor -- start 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.771+0000 7f7079333640 1 -- start start 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.771+0000 7f7079333640 10 monclient: init 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f70740a8970 0x7f707413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient(hunting): picked mon.noname-a con 0x7f70740a8970 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70740a9bd0 0x7f707413f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient(hunting): picked mon.noname-b con 0x7f70740a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70741421a0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70740aa950 con 0x7f70741421a0 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70740a90e0 con 0x7f70740a8970 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7074142d40 con 0x7f70740a9bd0 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.772+0000 7f7079333640 10 monclient(hunting): authenticate will time out at 1378.960523s 2023-12-07T23:46:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.801+0000 7fb3617fa640 1 -- 172.21.15.129:0/909364438 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb35c077b90 con 0x7fb36413f3b0 2023-12-07T23:46:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.801+0000 7fb3617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.801+0000 7fb3617fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.806+0000 7f70727fc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70740a9bd0 0x7f707413f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.817+0000 7f70727fc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70740a9bd0 0x7f707413f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55736/0 (socket says 172.21.15.129:55736) 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.817+0000 7f70727fc640 1 -- 172.21.15.129:0/3253714351 learned_addr learned my addr 172.21.15.129:0/3253714351 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.817+0000 7f70727fc640 10 monclient(hunting): get_auth_request con 0x7f70740a9bd0 auth_method 0 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.817+0000 7f70727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.817+0000 7f70727fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.817+0000 7f70727fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.818+0000 7f70727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.818+0000 7f70727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.818+0000 7f70727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2023-12-07T23:46:18.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 1 -- 172.21.15.129:0/3253714351 >> v2:172.21.15.129:3300/0 conn(0x7f70740a8970 msgr2=0x7f707413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 1 --2- 172.21.15.129:0/3253714351 >> v2:172.21.15.129:3300/0 conn(0x7f70740a8970 0x7f707413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 1 -- 172.21.15.129:0/3253714351 >> v1:172.21.15.129:6789/0 conn(0x7f70741421a0 legacy=0x7f7074142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 1 monclient: found mon.noname-b 2023-12-07T23:46:18.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.820+0000 7f70727fc640 1 -- 172.21.15.129:0/3253714351 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70740aabc0 con 0x7f70740a9bd0 2023-12-07T23:46:18.826 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.823+0000 7f70727fc640 1 --2- 172.21.15.129:0/3253714351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70740a9bd0 0x7f707413f3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7068009870 tx=0x7f706802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37ad2351311fa678 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 1 -- 172.21.15.129:0/3253714351 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068031740 con 0x7f70740a9bd0 2023-12-07T23:46:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: dump: 2023-12-07T23:46:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 1 -- 172.21.15.129:0/3253714351 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70680318a0 con 0x7f70740a9bd0 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 1 -- 172.21.15.129:0/3253714351 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068031c30 con 0x7f70740a9bd0 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7071ffb640 10 monclient: dump: 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.833 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.834 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7079333640 5 monclient: authenticate success, global_id 4289 2023-12-07T23:46:18.834 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7079333640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:18.834 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.828+0000 7f7079333640 4 set_mon_vals no callback set 2023-12-07T23:46:18.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.829+0000 7f7079333640 10 monclient: shutdown 2023-12-07T23:46:18.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.829+0000 7f7079333640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:18.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.829+0000 7f7079333640 1 -- 172.21.15.129:0/3253714351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70740a9bd0 msgr2=0x7f707413f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.829+0000 7f7079333640 1 --2- 172.21.15.129:0/3253714351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70740a9bd0 0x7f707413f3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7068009870 tx=0x7f706802b7e0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.831+0000 7f7079333640 1 -- 172.21.15.129:0/3253714351 shutdown_connections 2023-12-07T23:46:18.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.831+0000 7f7079333640 1 --2- 172.21.15.129:0/3253714351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70740a9bd0 0x7f707413f3b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.835 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.831+0000 7f7079333640 1 --2- 172.21.15.129:0/3253714351 >> v2:172.21.15.129:3300/0 conn(0x7f70740a8970 0x7f707413ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.831+0000 7f7079333640 1 -- 172.21.15.129:0/3253714351 >> 172.21.15.129:0/3253714351 conn(0x7f7074066dd0 msgr2=0x7f70740b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.831+0000 7f7079333640 1 -- 172.21.15.129:0/3253714351 shutdown_connections 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.831+0000 7f7079333640 1 -- 172.21.15.129:0/3253714351 wait complete. 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.832+0000 7f7079333640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.832+0000 7f7079333640 10 monclient: monmap: 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:18.836 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:18.837 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:18.837 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.833+0000 7f7079333640 1 librados: starting msgr at 2023-12-07T23:46:18.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.833+0000 7f7079333640 1 librados: starting objecter 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.833+0000 7f7079333640 1 Processor -- start 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.833+0000 7f7079333640 1 -- start start 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.833+0000 7f7079333640 1 librados: setting wanted keys 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.833+0000 7f7079333640 1 librados: calling monclient init 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.833+0000 7f7079333640 10 monclient: init 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.834+0000 7f7079333640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.834+0000 7f7079333640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.834+0000 7f7079333640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70740a8970 0x7f707413a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.834+0000 7f7079333640 10 monclient(hunting): picked mon.noname-b con 0x7f70740a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.834+0000 7f7079333640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70740a9bd0 0x7f707413ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.834+0000 7f7079333640 10 monclient(hunting): picked mon.noname-a con 0x7f70740a9bd0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:18.840 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.834+0000 7f7079333640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70741421a0 0x7f7074132480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 10 monclient(hunting): picked mon.noname-c con 0x7f70741421a0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70740abb40 con 0x7f70740a9bd0 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70740abd50 con 0x7f70740a8970 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707413b5d0 con 0x7f70741421a0 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f7079333640 10 monclient(hunting): authenticate will time out at 1379.023548s 2023-12-07T23:46:18.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f70737fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70741421a0 0x7f7074132480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f70737fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70741421a0 0x7f7074132480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55738/0 (socket says 172.21.15.129:55738) 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f70737fe640 1 -- 172.21.15.129:0/539324343 learned_addr learned my addr 172.21.15.129:0/539324343 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f70737fe640 10 monclient(hunting): get_auth_request con 0x7f70741421a0 auth_method 0 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f70737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f70737fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.835+0000 7f70737fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.836+0000 7f70737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.836+0000 7f70737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.836+0000 7f70737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.842 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-07T23:46:18.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2023-12-07T23:46:18.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 1 -- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70740a8970 msgr2=0x7f707413a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70740a8970 0x7f707413a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 1 -- 172.21.15.129:0/539324343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70740a9bd0 msgr2=0x7f707413ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70740a9bd0 0x7f707413ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 1 monclient: found mon.noname-c 2023-12-07T23:46:18.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.841+0000 7f70737fe640 1 -- 172.21.15.129:0/539324343 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70680094d0 con 0x7f70741421a0 2023-12-07T23:46:18.848 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.845+0000 7f70737fe640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70741421a0 0x7f7074132480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f706c00b9e0 tx=0x7f706c00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.848 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.845+0000 7f705b7fe640 10 client.?.objecter ms_handle_connect 0x7f70741421a0 2023-12-07T23:46:18.848 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.845+0000 7f705b7fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 1 -- 172.21.15.129:0/539324343 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c00caa0 con 0x7f70741421a0 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: dump: 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.855 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 1 -- 172.21.15.129:0/539324343 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c00cc00 con 0x7f70741421a0 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 1 -- 172.21.15.129:0/539324343 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c014440 con 0x7f70741421a0 2023-12-07T23:46:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f705b7fe640 10 monclient: dump: 2023-12-07T23:46:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.858 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 5 monclient: authenticate success, global_id 4292 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 10 monclient: _renew_subs 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707413bb50 con 0x7f70741421a0 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 10 monclient: _renew_subs 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70740ac420 con 0x7f70741421a0 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7079333640 1 librados: init done 2023-12-07T23:46:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f7070ff9640 4 set_mon_vals no callback set 2023-12-07T23:46:18.860 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f8930626640 10 monclient: get_monmap_and_config 2023-12-07T23:46:18.860 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.851+0000 7f7079333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.860 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f8930626640 10 monclient: build_initial_monmap 2023-12-07T23:46:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.851+0000 7f7079333640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.851+0000 7f7079333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.850+0000 7f8930626640 10 monclient: monmap: 2023-12-07T23:46:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:18.852088+0000 2023-12-07T23:46:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:18.852088+0000 2023-12-07T23:46:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:18.863 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.851+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70740ac6d0 con 0x7f70741421a0 2023-12-07T23:46:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.851+0000 7f8930626640 1 Processor -- start 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.851+0000 7f8930626640 1 -- start start 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.851+0000 7f8930626640 10 monclient: init 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f89280af180 0x7f89280a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): picked mon.noname-c con 0x7f89280af180 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280a89e0 0x7f8928071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): picked mon.noname-b con 0x7f89280a89e0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8928071a70 0x7f89280acc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): picked mon.noname-a con 0x7f8928071a70 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280af560 con 0x7f8928071a70 2023-12-07T23:46:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280726c0 con 0x7f89280af180 2023-12-07T23:46:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8928072b50 con 0x7f89280a89e0 2023-12-07T23:46:18.867 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:18.867 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.852+0000 7f8930626640 10 monclient(hunting): authenticate will time out at 1379.040701s 2023-12-07T23:46:18.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.949+0000 7f771644b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:18.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.949+0000 7f771644b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:18.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.949+0000 7f771644b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.949+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7710136c60 con 0x7f77100a8d30 2023-12-07T23:46:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.951+0000 7f65ee7fc640 1 -- 172.21.15.129:0/2365825212 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4011830 con 0x7f65f80a8970 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.951+0000 7f65ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.951+0000 7f65ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.951+0000 7f65ee7fc640 10 monclient: dump: 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.951+0000 7f65ee7fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.951+0000 7f65ee7fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65ee7fc640 1 -- 172.21.15.129:0/2365825212 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f4011990 con 0x7f65f80a8970 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 5 monclient: authenticate success, global_id 4378 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 10 monclient: _renew_subs 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f80ae960 con 0x7f65f80a8970 2023-12-07T23:46:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65ee7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65ee7fc640 1 -- 172.21.15.129:0/2365825212 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4011d40 con 0x7f65f80a8970 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65ee7fc640 10 monclient: dump: 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f65effff640 4 set_mon_vals no callback set 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 10 monclient: _renew_subs 2023-12-07T23:46:18.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f81db1d0 con 0x7f65f80a8970 2023-12-07T23:46:18.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 1 librados: init done 2023-12-07T23:46:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.952+0000 7f660021f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.953+0000 7f660021f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.953+0000 7f660021f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.953+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f804e840 con 0x7f65f80a8970 2023-12-07T23:46:18.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.964+0000 7f34727fc640 1 -- 172.21.15.129:0/1083168691 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3478005990 con 0x7f347c06a730 2023-12-07T23:46:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.964+0000 7f34727fc640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f346006c820 0x7f346006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.965+0000 7f348142f640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f346006c820 0x7f346006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.965+0000 7f348142f640 10 monclient: get_auth_request con 0x7f346006c820 auth_method 0 2023-12-07T23:46:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.965+0000 7f34727fc640 10 client.4375.objecter ms_handle_connect 0x7f346006c820 2023-12-07T23:46:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.965+0000 7f348142f640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f346006c820 0x7f346006ece0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f347c1d29a0 tx=0x7f346c007990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.967+0000 7f34727fc640 1 -- 172.21.15.129:0/1083168691 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f347807af40 con 0x7f347c06a730 2023-12-07T23:46:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.967+0000 7f34727fc640 10 client.4375.objecter ms_dispatch 0x7f347c000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.967+0000 7f34727fc640 3 client.4375.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.967+0000 7f34727fc640 3 client.4375.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.967+0000 7f34727fc640 20 client.4375.objecter dump_active .. 0 homeless 2023-12-07T23:46:18.980 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.976+0000 7f34727fc640 1 -- 172.21.15.129:0/1083168691 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3478076b80 con 0x7f347c06a730 2023-12-07T23:46:18.980 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.977+0000 7f34727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.977+0000 7f34727fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.978+0000 7f892e3c2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f89280af180 0x7f89280a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.981+0000 7f892e3c2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f89280af180 0x7f89280a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40832/0 (socket says 172.21.15.129:40832) 2023-12-07T23:46:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.981+0000 7f892e3c2640 1 -- 172.21.15.129:0/4127057579 learned_addr learned my addr 172.21.15.129:0/4127057579 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.981+0000 7f892e3c2640 10 monclient(hunting): get_auth_request con 0x7f89280af180 auth_method 0 2023-12-07T23:46:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.981+0000 7f892e3c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.981+0000 7f892e3c2640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.981+0000 7f892e3c2640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.982+0000 7f892e3c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.982+0000 7f892e3c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.982+0000 7f892e3c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2023-12-07T23:46:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 1 -- 172.21.15.129:0/4127057579 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280a89e0 msgr2=0x7f8928071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 1 --2- 172.21.15.129:0/4127057579 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280a89e0 0x7f8928071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 1 -- 172.21.15.129:0/4127057579 >> v2:172.21.15.129:3300/0 conn(0x7f8928071a70 msgr2=0x7f89280acc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 1 --2- 172.21.15.129:0/4127057579 >> v2:172.21.15.129:3300/0 conn(0x7f8928071a70 0x7f89280acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 10 monclient: _finish_hunting 0 2023-12-07T23:46:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 1 monclient: found mon.noname-c 2023-12-07T23:46:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.984+0000 7f892e3c2640 1 -- 172.21.15.129:0/4127057579 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89281431c0 con 0x7f89280af180 2023-12-07T23:46:18.990 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.987+0000 7f770cff9640 1 -- 172.21.15.129:0/456718907 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7710136c60 con 0x7f77100a8d30 2023-12-07T23:46:18.990 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.987+0000 7f770cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:18.990 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.987+0000 7f770cff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:18.991 INFO:teuthology.orchestra.run.smithi129.stdout:55834574852 2023-12-07T23:46:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.987+0000 7f892e3c2640 1 --2- 172.21.15.129:0/4127057579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89280af180 0x7f89280a84a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f89280b45c0 tx=0x7f892402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=501a323a8f47cd2c server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:18.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.989+0000 7f771644b640 10 librados: watch_flush enter 2023-12-07T23:46:18.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.989+0000 7f771644b640 10 librados: watch_flush exit 2023-12-07T23:46:18.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.989+0000 7f771644b640 20 client.4366.objecter shutdown clearing up homeless session... 2023-12-07T23:46:18.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.989+0000 7f771644b640 10 client.4366.objecter successfully canceled tick 2023-12-07T23:46:18.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f76fc06c820 msgr2=0x7f76fc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f76fc06c820 0x7f76fc06ece0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7704000f60 tx=0x7f7704009040 comp rx=0 tx=0).stop 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 10 monclient: shutdown 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77100a8d30 msgr2=0x7f7710138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77100a8d30 0x7f7710138200 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f770000c430 tx=0x7f770000c8f0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 shutdown_connections 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f76fc06c820 0x7f76fc06ece0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f771013f820 0x7f7710145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77100a8d30 0x7f7710138200 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 --2- 172.21.15.129:0/456718907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77100a8950 0x7f7710141960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 >> 172.21.15.129:0/456718907 conn(0x7f7710066cd0 msgr2=0x7f77100a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 shutdown_connections 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 -- 172.21.15.129:0/456718907 wait complete. 2023-12-07T23:46:18.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:18.990+0000 7f771644b640 1 librados: shutdown 2023-12-07T23:46:19.005 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.002+0000 7f51aa190640 10 monclient: get_monmap_and_config 2023-12-07T23:46:19.005 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.002+0000 7f51aa190640 10 monclient: build_initial_monmap 2023-12-07T23:46:19.005 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.002+0000 7f51aa190640 10 monclient: monmap: 2023-12-07T23:46:19.005 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:19.003276+0000 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:19.003276+0000 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:19.006 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2023-12-07T23:46:19.008 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 1 Processor -- start 2023-12-07T23:46:19.008 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 1 -- start start 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient: init 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a40a8970 0x7f51a413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient(hunting): picked mon.noname-b con 0x7f51a40a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51a40a9bd0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f51a413f3b0 0x7f51a41428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient(hunting): picked mon.noname-a con 0x7f51a413f3b0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a40aa950 con 0x7f51a40a9bd0 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a4142df0 con 0x7f51a413f3b0 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.003+0000 7f51aa190640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a413f840 con 0x7f51a40a8970 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.004+0000 7f51aa190640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.004+0000 7f51aa190640 10 monclient(hunting): authenticate will time out at 1379.191998s 2023-12-07T23:46:19.031 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.028+0000 7f82b57fa640 1 -- 172.21.15.129:0/2342258278 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82a40769b0 con 0x7f82b80b5610 2023-12-07T23:46:19.031 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.028+0000 7f82b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.031 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.028+0000 7f82b57fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:19.033 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.030+0000 7f51a3fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f51a413f3b0 0x7f51a41428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:19.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.032+0000 7f51a3fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f51a413f3b0 0x7f51a41428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43068/0 (socket says 172.21.15.129:43068) 2023-12-07T23:46:19.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.032+0000 7f51a3fff640 1 -- 172.21.15.129:0/776932992 learned_addr learned my addr 172.21.15.129:0/776932992 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:19.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.032+0000 7f51a3fff640 10 monclient(hunting): get_auth_request con 0x7f51a413f3b0 auth_method 0 2023-12-07T23:46:19.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.032+0000 7f51a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:19.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.032+0000 7f51a3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:19.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.032+0000 7f51a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.035+0000 7f51a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.035+0000 7f51a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.035+0000 7f51a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:19.042 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.038+0000 7f51a3fff640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2023-12-07T23:46:19.042 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.038+0000 7f51a3fff640 1 -- 172.21.15.129:0/776932992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a40a8970 msgr2=0x7f51a413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:19.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.038+0000 7f51a3fff640 1 --2- 172.21.15.129:0/776932992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a40a8970 0x7f51a413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.038+0000 7f51a3fff640 1 -- 172.21.15.129:0/776932992 >> v1:172.21.15.129:6789/0 conn(0x7f51a40a9bd0 legacy=0x7f51a41421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:19.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.038+0000 7f51a3fff640 10 monclient: _finish_hunting 0 2023-12-07T23:46:19.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.038+0000 7f51a3fff640 1 monclient: found mon.noname-a 2023-12-07T23:46:19.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.039+0000 7f51a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:19.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.039+0000 7f51a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:19.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.039+0000 7f51a3fff640 1 -- 172.21.15.129:0/776932992 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a40a90e0 con 0x7f51a413f3b0 2023-12-07T23:46:19.048 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.045+0000 7f51a3fff640 1 --2- 172.21.15.129:0/776932992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a413f3b0 0x7f51a41428b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f519c009e50 tx=0x7f519c02bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26be97113a6608f4 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:19.079 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.076+0000 7fb36af3e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:19.079 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.076+0000 7fb36af3e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:19.079 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.076+0000 7fb36af3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:19.079 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.076+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb364142a30 con 0x7fb36413f3b0 2023-12-07T23:46:19.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51a27fc640 1 -- 172.21.15.129:0/776932992 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c02c980 con 0x7f51a413f3b0 2023-12-07T23:46:19.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51a27fc640 10 monclient: dump: 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51a27fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51a27fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:19.121 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51aa190640 5 monclient: authenticate success, global_id 4353 2023-12-07T23:46:19.121 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.116+0000 7f51aa190640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51a27fc640 1 -- 172.21.15.129:0/776932992 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c02cae0 con 0x7f51a413f3b0 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51a27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51a27fc640 1 -- 172.21.15.129:0/776932992 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c0343f0 con 0x7f51a413f3b0 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51a27fc640 10 monclient: dump: 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.123 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51aa190640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.119+0000 7f51aa190640 4 set_mon_vals no callback set 2023-12-07T23:46:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 10 monclient: shutdown 2023-12-07T23:46:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 -- 172.21.15.129:0/776932992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a413f3b0 msgr2=0x7f51a41428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 --2- 172.21.15.129:0/776932992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a413f3b0 0x7f51a41428b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f519c009e50 tx=0x7f519c02bde0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 -- 172.21.15.129:0/776932992 shutdown_connections 2023-12-07T23:46:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 --2- 172.21.15.129:0/776932992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a413f3b0 0x7f51a41428b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 --2- 172.21.15.129:0/776932992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a40a8970 0x7f51a413ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 -- 172.21.15.129:0/776932992 >> 172.21.15.129:0/776932992 conn(0x7f51a4066dd0 msgr2=0x7f51a40b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 -- 172.21.15.129:0/776932992 shutdown_connections 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.120+0000 7f51aa190640 1 -- 172.21.15.129:0/776932992 wait complete. 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.121+0000 7f51aa190640 10 monclient: build_initial_monmap 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.121+0000 7f51aa190640 10 monclient: monmap: 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:19.126 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.121+0000 7f51aa190640 1 librados: starting msgr at 2023-12-07T23:46:19.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.121+0000 7f51aa190640 1 librados: starting objecter 2023-12-07T23:46:19.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.122+0000 7f51aa190640 1 Processor -- start 2023-12-07T23:46:19.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.122+0000 7f51aa190640 1 -- start start 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.122+0000 7f51aa190640 1 librados: setting wanted keys 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.122+0000 7f51aa190640 1 librados: calling monclient init 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.122+0000 7f51aa190640 10 monclient: init 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a40a8970 0x7f51a40b74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): picked mon.noname-a con 0x7f51a40a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51a40a9bd0 0x7f51a4139fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): picked mon.noname-b con 0x7f51a40a9bd0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a413f3b0 0x7f51a4133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): picked mon.noname-c con 0x7f51a413f3b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a413a690 con 0x7f51a40a8970 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4139490 con 0x7f51a40a9bd0 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4139600 con 0x7f51a413f3b0 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51aa190640 10 monclient(hunting): authenticate will time out at 1379.311390s 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.123+0000 7f51a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a40a8970 0x7f51a40b74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a40a8970 0x7f51a40b74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43076/0 (socket says 172.21.15.129:43076) 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 1 -- 172.21.15.129:0/3448015958 learned_addr learned my addr 172.21.15.129:0/3448015958 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 10 monclient(hunting): get_auth_request con 0x7f51a40a8970 auth_method 0 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.124+0000 7f51a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:19.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2023-12-07T23:46:19.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 1 -- 172.21.15.129:0/3448015958 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a413f3b0 msgr2=0x7f51a4133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a413f3b0 0x7f51a4133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 1 -- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51a40a9bd0 msgr2=0x7f51a4139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51a40a9bd0 0x7f51a4139fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 1 monclient: found mon.noname-a 2023-12-07T23:46:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.127+0000 7f51a37fe640 1 -- 172.21.15.129:0/3448015958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c009b30 con 0x7f51a40a8970 2023-12-07T23:46:19.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.130+0000 7f51a37fe640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a40a8970 0x7f51a40b74c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f519400b4a0 tx=0x7f519400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:19.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.130+0000 7f518bfff640 10 client.?.objecter ms_handle_connect 0x7f51a40a8970 2023-12-07T23:46:19.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.130+0000 7f518bfff640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:19.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 1 -- 172.21.15.129:0/3448015958 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194010610 con 0x7f51a40a8970 2023-12-07T23:46:19.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:19.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: dump: 2023-12-07T23:46:19.141 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.141 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.141 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.142 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.142 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.142 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.142 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.142 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.142 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.143 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: _finish_auth 0 2023-12-07T23:46:19.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: _check_auth_tickets 2023-12-07T23:46:19.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:19.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 1 -- 172.21.15.129:0/3448015958 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5194010770 con 0x7f51a40a8970 2023-12-07T23:46:19.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 5 monclient: authenticate success, global_id 4356 2023-12-07T23:46:19.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 10 monclient: _renew_subs 2023-12-07T23:46:19.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:19.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a4133db0 con 0x7f51a40a8970 2023-12-07T23:46:19.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:19.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 1 -- 172.21.15.129:0/3448015958 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194010b20 con 0x7f51a40a8970 2023-12-07T23:46:19.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:19.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f518bfff640 10 monclient: dump: 2023-12-07T23:46:19.145 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.145 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.145 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.145 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.145 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.146 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.146 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.146 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.146 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.146 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51a1ffb640 4 set_mon_vals no callback set 2023-12-07T23:46:19.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:19.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 10 monclient: _renew_subs 2023-12-07T23:46:19.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:19.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a41342c0 con 0x7f51a40a8970 2023-12-07T23:46:19.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.135+0000 7f51aa190640 1 librados: init done 2023-12-07T23:46:19.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.136+0000 7f51aa190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.136+0000 7f51aa190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.148 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.136+0000 7f51aa190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:19.148 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.136+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a413a500 con 0x7f51a40a8970 2023-12-07T23:46:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.173+0000 7f518bfff640 1 -- 172.21.15.129:0/3448015958 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5194010c80 con 0x7f51a40a8970 2023-12-07T23:46:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.173+0000 7f518bfff640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f519006c850 0x7f519006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.173+0000 7f51a2ffd640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f519006c850 0x7f519006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.173+0000 7f51a2ffd640 10 monclient: get_auth_request con 0x7f519006c850 auth_method 0 2023-12-07T23:46:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.174+0000 7f518bfff640 10 client.4356.objecter ms_handle_connect 0x7f519006c850 2023-12-07T23:46:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.174+0000 7f51a2ffd640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f519006c850 0x7f519006ed10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f51a413fd40 tx=0x7f51980079b0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.199+0000 7f3483e94640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.199+0000 7f3483e94640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.199+0000 7f3483e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.199+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3464005600 con 0x7f347c06a730 2023-12-07T23:46:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 1 -- 172.21.15.129:0/4127057579 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89240317e0 con 0x7f89280af180 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: dump: 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.221 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: _finish_auth 0 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: _check_auth_tickets 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 1 -- 172.21.15.129:0/4127057579 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924031940 con 0x7f89280af180 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 5 monclient: authenticate success, global_id 4381 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 1 -- 172.21.15.129:0/4127057579 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924031cd0 con 0x7f89280af180 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f892d3c0640 10 monclient: dump: 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.222 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:19.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 4 set_mon_vals no callback set 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 10 monclient: shutdown 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 -- 172.21.15.129:0/4127057579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89280af180 msgr2=0x7f89280a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 --2- 172.21.15.129:0/4127057579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89280af180 0x7f89280a84a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f89280b45c0 tx=0x7f892402b880 comp rx=0 tx=0).stop 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 -- 172.21.15.129:0/4127057579 shutdown_connections 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 --2- 172.21.15.129:0/4127057579 >> v2:172.21.15.129:3300/0 conn(0x7f8928071a70 0x7f89280acc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 --2- 172.21.15.129:0/4127057579 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280a89e0 0x7f8928071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 --2- 172.21.15.129:0/4127057579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89280af180 0x7f89280a84a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 -- 172.21.15.129:0/4127057579 >> 172.21.15.129:0/4127057579 conn(0x7f892806aaf0 msgr2=0x7f89280b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.218+0000 7f8930626640 1 -- 172.21.15.129:0/4127057579 shutdown_connections 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.219+0000 7f8930626640 1 -- 172.21.15.129:0/4127057579 wait complete. 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.219+0000 7f8930626640 10 monclient: build_initial_monmap 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.219+0000 7f8930626640 10 monclient: monmap: 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.219+0000 7f8930626640 1 librados: starting msgr at 2023-12-07T23:46:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.219+0000 7f8930626640 1 librados: starting objecter 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 Processor -- start 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 -- start start 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 librados: setting wanted keys 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 librados: calling monclient init 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient: init 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8928071a70 0x7f89280ad200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): picked mon.noname-b con 0x7f8928071a70 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89280a89e0 0x7f89280708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): picked mon.noname-a con 0x7f89280a89e0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280af180 0x7f8928070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): picked mon.noname-c con 0x7f89280af180 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89281380a0 con 0x7f89280a89e0 2023-12-07T23:46:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8928138380 con 0x7f8928071a70 2023-12-07T23:46:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8928138660 con 0x7f89280af180 2023-12-07T23:46:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.220+0000 7f8930626640 10 monclient(hunting): authenticate will time out at 1379.408598s 2023-12-07T23:46:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.223+0000 7f82a2ffd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:19.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.223+0000 7f82a2ffd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:19.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.223+0000 7f82a2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:19.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.223+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8290005600 con 0x7f82b80b5610 2023-12-07T23:46:19.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.225+0000 7f705b7fe640 1 -- 172.21.15.129:0/539324343 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f706c01d3b0 con 0x7f70741421a0 2023-12-07T23:46:19.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.225+0000 7f705b7fe640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f706006c820 0x7f706006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.225+0000 7f7072ffd640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f706006c820 0x7f706006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:19.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.225+0000 7f7072ffd640 10 monclient: get_auth_request con 0x7f706006c820 auth_method 0 2023-12-07T23:46:19.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.226+0000 7f7072ffd640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f706006c820 0x7f706006ece0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7064009710 tx=0x7f7064006c90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:19.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.226+0000 7f705b7fe640 10 client.4292.objecter ms_handle_connect 0x7f706006c820 2023-12-07T23:46:19.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.230+0000 7f65ee7fc640 1 -- 172.21.15.129:0/2365825212 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f65f401b460 con 0x7f65f80a8970 2023-12-07T23:46:19.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.231+0000 7f65ee7fc640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f65dc06c870 0x7f65dc06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.231+0000 7f65fd7ba640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f65dc06c870 0x7f65dc06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:19.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.231+0000 7f65fd7ba640 10 monclient: get_auth_request con 0x7f65dc06c870 auth_method 0 2023-12-07T23:46:19.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.231+0000 7f65ee7fc640 10 client.4378.objecter ms_handle_connect 0x7f65dc06c870 2023-12-07T23:46:19.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.231+0000 7f65fd7ba640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f65dc06c870 0x7f65dc06ed30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f65f8141c00 tx=0x7f65e8007990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:19.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.235+0000 7f65ee7fc640 1 -- 172.21.15.129:0/2365825212 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f65f4059330 con 0x7f65f80a8970 2023-12-07T23:46:19.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.235+0000 7f65ee7fc640 10 client.4378.objecter ms_dispatch 0x7f65f8000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:19.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.235+0000 7f65ee7fc640 3 client.4378.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:19.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.235+0000 7f65ee7fc640 3 client.4378.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:19.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.235+0000 7f65ee7fc640 20 client.4378.objecter dump_active .. 0 homeless 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.240+0000 7f705b7fe640 1 -- 172.21.15.129:0/539324343 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f706c058e90 con 0x7f70741421a0 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.240+0000 7f705b7fe640 10 client.4292.objecter ms_dispatch 0x7f7074000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.240+0000 7f705b7fe640 3 client.4292.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.240+0000 7f705b7fe640 3 client.4292.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.241+0000 7f705b7fe640 20 client.4292.objecter dump_active .. 0 homeless 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.241+0000 7f705b7fe640 1 -- 172.21.15.129:0/539324343 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f706c054ad0 con 0x7f70741421a0 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.241+0000 7f705b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.241+0000 7f705b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:19.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.243+0000 7fb3617fa640 1 -- 172.21.15.129:0/909364438 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb35c07b1b0 con 0x7fb36413f3b0 2023-12-07T23:46:19.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.243+0000 7fb3617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:19.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.243+0000 7fb3617fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:19.246 INFO:teuthology.orchestra.run.smithi129.stdout:68719476741 2023-12-07T23:46:19.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.243+0000 7f892ebc3640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280af180 0x7f8928070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:19.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280af180 0x7f8928070df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55762/0 (socket says 172.21.15.129:55762) 2023-12-07T23:46:19.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 1 -- 172.21.15.129:0/3072904859 learned_addr learned my addr 172.21.15.129:0/3072904859 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:19.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 10 monclient(hunting): get_auth_request con 0x7f89280af180 auth_method 0 2023-12-07T23:46:19.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:19.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:19.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:19.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:19.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:19.248 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.244+0000 7f892ebc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:19.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 10 librados: watch_flush enter 2023-12-07T23:46:19.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 10 librados: watch_flush exit 2023-12-07T23:46:19.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 20 client.4286.objecter shutdown clearing up homeless session... 2023-12-07T23:46:19.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 10 client.4286.objecter successfully canceled tick 2023-12-07T23:46:19.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb35006c8a0 msgr2=0x7fb35006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb35006c8a0 0x7fb35006ed60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb354009710 tx=0x7fb354006c90 comp rx=0 tx=0).stop 2023-12-07T23:46:19.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 10 monclient: shutdown 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 msgr2=0x7fb3641d7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641d7950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009e50 tx=0x7fb35c004a40 comp rx=0 tx=0).stop 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 shutdown_connections 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb35006c8a0 0x7fb35006ed60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb36413f3b0 0x7fb3641d7950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640a9bd0 0x7fb3641d7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 --2- 172.21.15.129:0/909364438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3640a8970 0x7fb364137d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 >> 172.21.15.129:0/909364438 conn(0x7fb364066dd0 msgr2=0x7fb3640b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 shutdown_connections 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 -- 172.21.15.129:0/909364438 wait complete. 2023-12-07T23:46:19.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7fb36af3e640 1 librados: shutdown 2023-12-07T23:46:19.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7f65ee7fc640 1 -- 172.21.15.129:0/2365825212 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65f4058a00 con 0x7f65f80a8970 2023-12-07T23:46:19.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7f65ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.245+0000 7f65ee7fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:19.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.247+0000 7f892ebc3640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-12-07T23:46:19.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.247+0000 7f892ebc3640 1 -- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8928071a70 msgr2=0x7f89280ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8928071a70 0x7f89280ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 1 -- 172.21.15.129:0/3072904859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89280a89e0 msgr2=0x7f89280708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89280a89e0 0x7f89280708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 10 monclient: _finish_hunting 0 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 1 monclient: found mon.noname-c 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:19.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.248+0000 7f892ebc3640 1 -- 172.21.15.129:0/3072904859 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240094d0 con 0x7f89280af180 2023-12-07T23:46:19.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.252+0000 7f892ebc3640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280af180 0x7f8928070df0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8918009fd0 tx=0x7f891800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:19.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.252+0000 7f891effd640 10 client.?.objecter ms_handle_connect 0x7f89280af180 2023-12-07T23:46:19.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.252+0000 7f891effd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:19.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f891effd640 1 -- 172.21.15.129:0/3072904859 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918011830 con 0x7f89280af180 2023-12-07T23:46:19.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f891effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f891effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:19.259 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f891effd640 10 monclient: dump: 2023-12-07T23:46:19.259 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.260 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.261 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.261 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f891effd640 10 monclient: _finish_auth 0 2023-12-07T23:46:19.261 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f891effd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:19.261 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f891effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:19.261 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 5 monclient: authenticate success, global_id 4295 2023-12-07T23:46:19.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 10 monclient: _renew_subs 2023-12-07T23:46:19.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:19.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8928070310 con 0x7f89280af180 2023-12-07T23:46:19.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:19.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 10 monclient: _renew_subs 2023-12-07T23:46:19.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:19.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8928138c60 con 0x7f89280af180 2023-12-07T23:46:19.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.255+0000 7f8930626640 1 librados: init done 2023-12-07T23:46:19.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.257+0000 7f8930626640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.257+0000 7f8930626640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.257+0000 7f8930626640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:19.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.257+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8928070550 con 0x7f89280af180 2023-12-07T23:46:19.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.3 2023-12-07T23:46:19.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.276+0000 7f891effd640 1 -- 172.21.15.129:0/3072904859 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918011990 con 0x7f89280af180 2023-12-07T23:46:19.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.276+0000 7f891effd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:19.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.276+0000 7f891effd640 1 -- 172.21.15.129:0/3072904859 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918009280 con 0x7f89280af180 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.276+0000 7f891effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.276+0000 7f891effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.276+0000 7f891effd640 10 monclient: dump: 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:19.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.276+0000 7f892cbbf640 4 set_mon_vals no callback set 2023-12-07T23:46:19.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.277+0000 7f891effd640 1 -- 172.21.15.129:0/3072904859 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f891801b460 con 0x7f89280af180 2023-12-07T23:46:19.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.277+0000 7f891effd640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f890c06c820 0x7f890c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:19.281 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.278+0000 7f892e3c2640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f890c06c820 0x7f890c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:19.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.278+0000 7f892e3c2640 10 monclient: get_auth_request con 0x7f890c06c820 auth_method 0 2023-12-07T23:46:19.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.278+0000 7f891effd640 10 client.4295.objecter ms_handle_connect 0x7f890c06c820 2023-12-07T23:46:19.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.278+0000 7f892e3c2640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f890c06c820 0x7f890c06ece0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8924036000 tx=0x7f8924030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:19.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.279+0000 7f891effd640 1 -- 172.21.15.129:0/3072904859 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f89180592c0 con 0x7f89280af180 2023-12-07T23:46:19.282 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.279+0000 7f891effd640 10 client.4295.objecter ms_dispatch 0x7f8928000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:19.283 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.279+0000 7f891effd640 3 client.4295.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:19.283 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.279+0000 7f891effd640 3 client.4295.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:19.283 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.279+0000 7f891effd640 20 client.4295.objecter dump_active .. 0 homeless 2023-12-07T23:46:19.304 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.301+0000 7f34727fc640 1 -- 172.21.15.129:0/1083168691 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f347807a1a0 con 0x7f347c06a730 2023-12-07T23:46:19.304 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.301+0000 7f34727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:19.304 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.301+0000 7f34727fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:19.304 INFO:teuthology.orchestra.run.smithi129.stdout:68719476741 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 10 librados: watch_flush enter 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 10 librados: watch_flush exit 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 20 client.4375.objecter shutdown clearing up homeless session... 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 10 client.4375.objecter successfully canceled tick 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f346006c820 msgr2=0x7f346006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f346006c820 0x7f346006ece0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f347c1d29a0 tx=0x7f346c007990 comp rx=0 tx=0).stop 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 10 monclient: shutdown 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 msgr2=0x7f347c149860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c149860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3478000c00 tx=0x7f34780046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 shutdown_connections 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f346006c820 0x7f346006ece0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c146de0 0x7f347c1df4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06fd90 0x7f347c1d0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 --2- 172.21.15.129:0/1083168691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f347c06a730 0x7f347c149860 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 >> 172.21.15.129:0/1083168691 conn(0x7f347c0b4a80 msgr2=0x7f347c073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 shutdown_connections 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 -- 172.21.15.129:0/1083168691 wait complete. 2023-12-07T23:46:19.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.303+0000 7f3483e94640 1 librados: shutdown 2023-12-07T23:46:19.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.2 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.431+0000 7f518bfff640 1 -- 172.21.15.129:0/3448015958 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5194058b00 con 0x7f51a40a8970 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.431+0000 7f518bfff640 10 client.4356.objecter ms_dispatch 0x7f51a4000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.431+0000 7f518bfff640 3 client.4356.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.431+0000 7f518bfff640 3 client.4356.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.431+0000 7f518bfff640 20 client.4356.objecter dump_active .. 0 homeless 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.431+0000 7f518bfff640 1 -- 172.21.15.129:0/3448015958 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5194054740 con 0x7f51a40a8970 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.432+0000 7f518bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.432+0000 7f518bfff640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:19.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.442+0000 7f82b57fa640 1 -- 172.21.15.129:0/2342258278 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f82a4079fd0 con 0x7f82b80b5610 2023-12-07T23:46:19.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.442+0000 7f82b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:19.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.442+0000 7f82b57fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:19.446 INFO:teuthology.orchestra.run.smithi129.stdout:55834574851 2023-12-07T23:46:19.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 10 librados: watch_flush enter 2023-12-07T23:46:19.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 10 librados: watch_flush exit 2023-12-07T23:46:19.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 20 client.4350.objecter shutdown clearing up homeless session... 2023-12-07T23:46:19.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 10 client.4350.objecter successfully canceled tick 2023-12-07T23:46:19.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f829c06c870 msgr2=0x7f829c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f829c06c870 0x7f829c06ed30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009670 tx=0x7f82ac006bf0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 10 monclient: shutdown 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 msgr2=0x7f82b81dab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81dab80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009870 tx=0x7f82a40046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 shutdown_connections 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f829c06c870 0x7f829c06ed30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82b81db0c0 0x7f82b812f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b80b5610 0x7f82b81dab80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 --2- 172.21.15.129:0/2342258278 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f82b80a8950 0x7f82b813b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.445+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 >> 172.21.15.129:0/2342258278 conn(0x7f82b8066cd0 msgr2=0x7f82b813e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.446+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 shutdown_connections 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.446+0000 7f82a2ffd640 1 -- 172.21.15.129:0/2342258278 wait complete. 2023-12-07T23:46:19.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.446+0000 7f82a2ffd640 1 librados: shutdown 2023-12-07T23:46:19.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.4 2023-12-07T23:46:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.542+0000 7f891effd640 1 -- 172.21.15.129:0/3072904859 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8918058990 con 0x7f89280af180 2023-12-07T23:46:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.542+0000 7f891effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.542+0000 7f891effd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:19.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.562+0000 7f7079333640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T23:46:19.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.562+0000 7f7079333640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T23:46:19.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.562+0000 7f7079333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:19.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.562+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f707404e840 con 0x7f70741421a0 2023-12-07T23:46:19.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.575+0000 7f660021f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T23:46:19.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.575+0000 7f660021f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T23:46:19.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.575+0000 7f660021f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:19.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.575+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f65f80725e0 con 0x7f65f80a8970 2023-12-07T23:46:19.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.578+0000 7f65ee7fc640 1 -- 172.21.15.129:0/2365825212 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65f4058590 con 0x7f65f80a8970 2023-12-07T23:46:19.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.578+0000 7f65ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T23:46:19.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.578+0000 7f65ee7fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:19.581 INFO:teuthology.orchestra.run.smithi129.stdout:55834574854 2023-12-07T23:46:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 10 librados: watch_flush enter 2023-12-07T23:46:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 10 librados: watch_flush exit 2023-12-07T23:46:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 20 client.4378.objecter shutdown clearing up homeless session... 2023-12-07T23:46:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 10 client.4378.objecter successfully canceled tick 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f65dc06c870 msgr2=0x7f65dc06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f65dc06c870 0x7f65dc06ed30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f65f8141c00 tx=0x7f65e8007990 comp rx=0 tx=0).stop 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 10 monclient: shutdown 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f80a8970 msgr2=0x7f65f813a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.580+0000 7f660021f640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f80a8970 0x7f65f813a320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f65f400b4a0 tx=0x7f65f400b960 comp rx=0 tx=0).stop 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 shutdown_connections 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f65dc06c870 0x7f65dc06ed30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65f813f3b0 0x7f65f81d6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65f80a9bd0 0x7f65f81da320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 --2- 172.21.15.129:0/2365825212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f80a8970 0x7f65f813a320 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 >> 172.21.15.129:0/2365825212 conn(0x7f65f8066dd0 msgr2=0x7f65f813be10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 shutdown_connections 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 -- 172.21.15.129:0/2365825212 wait complete. 2023-12-07T23:46:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.581+0000 7f660021f640 1 librados: shutdown 2023-12-07T23:46:19.596 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2023-12-07T23:46:19.596 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.661+0000 7f705b7fe640 1 -- 172.21.15.129:0/539324343 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f706c0580f0 con 0x7f70741421a0 2023-12-07T23:46:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.661+0000 7f705b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T23:46:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.661+0000 7f705b7fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:19.665 INFO:teuthology.orchestra.run.smithi129.stdout:60129542148 2023-12-07T23:46:19.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.663+0000 7f7079333640 10 librados: watch_flush enter 2023-12-07T23:46:19.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.663+0000 7f7079333640 10 librados: watch_flush exit 2023-12-07T23:46:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.663+0000 7f7079333640 20 client.4292.objecter shutdown clearing up homeless session... 2023-12-07T23:46:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.663+0000 7f7079333640 10 client.4292.objecter successfully canceled tick 2023-12-07T23:46:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.663+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f706006c820 msgr2=0x7f706006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f706006c820 0x7f706006ece0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7064009710 tx=0x7f7064006c90 comp rx=0 tx=0).stop 2023-12-07T23:46:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 10 monclient: shutdown 2023-12-07T23:46:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70741421a0 msgr2=0x7f7074132480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70741421a0 0x7f7074132480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f706c00b9e0 tx=0x7f706c00bea0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 shutdown_connections 2023-12-07T23:46:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f706006c820 0x7f706006ece0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70741421a0 0x7f7074132480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70740a9bd0 0x7f707413ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 --2- 172.21.15.129:0/539324343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70740a8970 0x7f707413a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 >> 172.21.15.129:0/539324343 conn(0x7f7074066dd0 msgr2=0x7f70741346b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 shutdown_connections 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 -- 172.21.15.129:0/539324343 wait complete. 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.664+0000 7f7079333640 1 librados: shutdown 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.668+0000 7f51aa190640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.668+0000 7f51aa190640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.668+0000 7f51aa190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:19.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.668+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f51a414c760 con 0x7f51a40a8970 2023-12-07T23:46:19.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.674+0000 7f518bfff640 1 -- 172.21.15.129:0/3448015958 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51a414c760 con 0x7f51a40a8970 2023-12-07T23:46:19.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.674+0000 7f518bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T23:46:19.678 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.674+0000 7f518bfff640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:19.678 INFO:teuthology.orchestra.run.smithi129.stdout:68719476742 2023-12-07T23:46:19.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.1 2023-12-07T23:46:19.679 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:19.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 10 librados: watch_flush enter 2023-12-07T23:46:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 10 librados: watch_flush exit 2023-12-07T23:46:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 20 client.4356.objecter shutdown clearing up homeless session... 2023-12-07T23:46:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 10 client.4356.objecter successfully canceled tick 2023-12-07T23:46:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f519006c850 msgr2=0x7f519006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f519006c850 0x7f519006ed10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f51a413fd40 tx=0x7f51980079b0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 10 monclient: shutdown 2023-12-07T23:46:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a40a8970 msgr2=0x7f51a40b74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a40a8970 0x7f51a40b74c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f519400b4a0 tx=0x7f519400b960 comp rx=0 tx=0).stop 2023-12-07T23:46:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 shutdown_connections 2023-12-07T23:46:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f519006c850 0x7f519006ed10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51a413f3b0 0x7f51a4133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51a40a9bd0 0x7f51a4139fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 --2- 172.21.15.129:0/3448015958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51a40a8970 0x7f51a40b74c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 >> 172.21.15.129:0/3448015958 conn(0x7f51a4066dd0 msgr2=0x7f51a41370f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 shutdown_connections 2023-12-07T23:46:19.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 -- 172.21.15.129:0/3448015958 wait complete. 2023-12-07T23:46:19.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.677+0000 7f51aa190640 1 librados: shutdown 2023-12-07T23:46:19.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.0 2023-12-07T23:46:19.695 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.743+0000 7f8930626640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T23:46:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.743+0000 7f8930626640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T23:46:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.743+0000 7f8930626640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.743+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8928070550 con 0x7f89280af180 2023-12-07T23:46:19.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.747+0000 7f891effd640 1 -- 172.21.15.129:0/3072904859 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8928070550 con 0x7f89280af180 2023-12-07T23:46:19.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.747+0000 7f891effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T23:46:19.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.747+0000 7f891effd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:19.750 INFO:teuthology.orchestra.run.smithi129.stdout:55834574853 2023-12-07T23:46:19.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 10 librados: watch_flush enter 2023-12-07T23:46:19.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 10 librados: watch_flush exit 2023-12-07T23:46:19.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 20 client.4295.objecter shutdown clearing up homeless session... 2023-12-07T23:46:19.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 10 client.4295.objecter successfully canceled tick 2023-12-07T23:46:19.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f890c06c820 msgr2=0x7f890c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f890c06c820 0x7f890c06ece0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8924036000 tx=0x7f8924030000 comp rx=0 tx=0).stop 2023-12-07T23:46:19.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 10 monclient: shutdown 2023-12-07T23:46:19.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:19.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280af180 msgr2=0x7f8928070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:19.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280af180 0x7f8928070df0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8918009fd0 tx=0x7f891800b8e0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 shutdown_connections 2023-12-07T23:46:19.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f890c06c820 0x7f890c06ece0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f89280af180 0x7f8928070df0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89280a89e0 0x7f89280708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 --2- 172.21.15.129:0/3072904859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8928071a70 0x7f89280ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:19.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 >> 172.21.15.129:0/3072904859 conn(0x7f892806aaf0 msgr2=0x7f8928141d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:19.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 shutdown_connections 2023-12-07T23:46:19.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 -- 172.21.15.129:0/3072904859 wait complete. 2023-12-07T23:46:19.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:19.749+0000 7f8930626640 1 librados: shutdown 2023-12-07T23:46:19.764 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.6 2023-12-07T23:46:19.764 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:20.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.108+0000 7f9a18894640 10 monclient: get_monmap_and_config 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.108+0000 7f9a18894640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.108+0000 7f9a18894640 10 monclient: monmap: 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:20.109214+0000 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:20.109214+0000 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:20.112 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.108+0000 7f9a18894640 1 Processor -- start 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.108+0000 7f9a18894640 1 -- start start 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.108+0000 7f9a18894640 10 monclient: init 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9a140a8950 0x7f9a140a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient(hunting): picked mon.noname-c con 0x7f9a140a8950 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a140b5610 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a1413f110 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a14141e30 con 0x7f9a140b5610 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140aa890 con 0x7f9a1413f110 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a140b5a30 con 0x7f9a140a8950 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.109+0000 7f9a18894640 10 monclient(hunting): authenticate will time out at 1380.297416s 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.110+0000 7f9a11d9b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f9a140b5610 0x7f9a141425a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43204/0 (socket says 172.21.15.129:43204) 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.110+0000 7f9a11d9b640 1 -- 172.21.15.129:0/1521195663 learned_addr learned my addr 172.21.15.129:0/1521195663 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.110+0000 7f9a1259c640 1 --2- 172.21.15.129:0/1521195663 >> v2:172.21.15.184:3300/0 conn(0x7f9a140a8950 0x7f9a140a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.110+0000 7f9a1259c640 10 monclient(hunting): get_auth_request con 0x7f9a140a8950 auth_method 0 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.110+0000 7f9a1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.110+0000 7f9a1259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.110+0000 7f9a1259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.111+0000 7f9a1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.111+0000 7f9a1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.111+0000 7f9a1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.112+0000 7f9a1159a640 1 -- 172.21.15.129:0/1521195663 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644998812 0 0) 0x7f9a14141e30 con 0x7f9a140b5610 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.112+0000 7f9a1159a640 10 monclient(hunting): _negotiate 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.112+0000 7f9a1159a640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.112+0000 7f9a1159a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.112+0000 7f9a1159a640 10 monclient(hunting): my global_id is 4359 2023-12-07T23:46:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.112+0000 7f9a1159a640 1 -- 172.21.15.129:0/1521195663 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99fc003520 con 0x7f9a140b5610 2023-12-07T23:46:20.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2023-12-07T23:46:20.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 1 -- 172.21.15.129:0/1521195663 >> v1:172.21.15.184:6789/0 conn(0x7f9a1413f110 legacy=0x7f9a1414c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 1 -- 172.21.15.129:0/1521195663 >> v1:172.21.15.129:6789/0 conn(0x7f9a140b5610 legacy=0x7f9a141425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 1 monclient: found mon.noname-c 2023-12-07T23:46:20.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.117 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.117 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.113+0000 7f9a1259c640 1 -- 172.21.15.129:0/1521195663 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a14142cb0 con 0x7f9a140a8950 2023-12-07T23:46:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.118+0000 7f9a1259c640 1 --2- 172.21.15.129:0/1521195663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a140a8d30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009d90 tx=0x7f9a0402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9ab3a4c0a572dbb server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 1 -- 172.21.15.129:0/1521195663 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0402cd00 con 0x7f9a140a8950 2023-12-07T23:46:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: dump: 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 1 -- 172.21.15.129:0/1521195663 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0402ce60 con 0x7f9a140a8950 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 1 -- 172.21.15.129:0/1521195663 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04034660 con 0x7f9a140a8950 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a18894640 5 monclient: authenticate success, global_id 4384 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a18894640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a18894640 4 set_mon_vals no callback set 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.121+0000 7f9a1159a640 10 monclient: dump: 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.127 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.127 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.127 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.127 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.127 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 10 monclient: shutdown 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 1 -- 172.21.15.129:0/1521195663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 msgr2=0x7f9a140a8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 1 --2- 172.21.15.129:0/1521195663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a140a8d30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009d90 tx=0x7f9a0402bf50 comp rx=0 tx=0).stop 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 1 -- 172.21.15.129:0/1521195663 shutdown_connections 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 1 --2- 172.21.15.129:0/1521195663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a140a8d30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 1 -- 172.21.15.129:0/1521195663 >> 172.21.15.129:0/1521195663 conn(0x7f9a14066cd0 msgr2=0x7f9a140b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 1 -- 172.21.15.129:0/1521195663 shutdown_connections 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.122+0000 7f9a18894640 1 -- 172.21.15.129:0/1521195663 wait complete. 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 10 monclient: monmap: 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 1 librados: starting msgr at 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 1 librados: starting objecter 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 1 Processor -- start 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 1 -- start start 2023-12-07T23:46:20.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 1 librados: setting wanted keys 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 1 librados: calling monclient init 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.123+0000 7f9a18894640 10 monclient: init 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a1413a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): picked mon.noname-b con 0x7f9a140a8950 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a140b5610 0x7f9a14144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): picked mon.noname-c con 0x7f9a140b5610 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1413f110 0x7f9a141453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): picked mon.noname-a con 0x7f9a1413f110 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14141080 con 0x7f9a1413f110 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1413a890 con 0x7f9a140a8950 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14138140 con 0x7f9a140b5610 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a18894640 10 monclient(hunting): authenticate will time out at 1380.312493s 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.124+0000 7f9a1259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a1413a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a11d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a140b5610 0x7f9a14144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a12d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1413f110 0x7f9a141453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a1413a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40868/0 (socket says 172.21.15.129:40868) 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 1 -- 172.21.15.129:0/3235718773 learned_addr learned my addr 172.21.15.129:0/3235718773 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 10 monclient(hunting): get_auth_request con 0x7f9a140a8950 auth_method 0 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a11d9b640 10 monclient(hunting): get_auth_request con 0x7f9a140b5610 auth_method 0 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a11d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a11d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a12d9d640 10 monclient(hunting): get_auth_request con 0x7f9a1413f110 auth_method 0 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a12d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a12d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.125+0000 7f9a1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.126+0000 7f9a12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.126+0000 7f9a12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.126+0000 7f9a12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.126+0000 7f9a11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.126+0000 7f9a11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.126+0000 7f9a11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.135 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2023-12-07T23:46:20.135 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 1 -- 172.21.15.129:0/3235718773 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a140b5610 msgr2=0x7f9a14144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.135 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a140b5610 0x7f9a14144b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.135 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 1 -- 172.21.15.129:0/3235718773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1413f110 msgr2=0x7f9a141453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.135 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1413f110 0x7f9a141453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.135 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 1 monclient: found mon.noname-b 2023-12-07T23:46:20.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.128+0000 7f9a1259c640 1 -- 172.21.15.129:0/3235718773 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04009a70 con 0x7f9a140a8950 2023-12-07T23:46:20.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.130+0000 7f9a1259c640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a1413a350 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009d90 tx=0x7f9a04004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.130+0000 7f9a02ffd640 10 client.?.objecter ms_handle_connect 0x7f9a140a8950 2023-12-07T23:46:20.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.130+0000 7f9a02ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 1 -- 172.21.15.129:0/3235718773 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0403e610 con 0x7f9a140a8950 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: dump: 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.137 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 1 -- 172.21.15.129:0/3235718773 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0403e770 con 0x7f9a140a8950 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a18894640 5 monclient: authenticate success, global_id 4390 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 1 -- 172.21.15.129:0/3235718773 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0403eb20 con 0x7f9a140a8950 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a18894640 10 monclient: _renew_subs 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a18894640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a14138420 con 0x7f9a140a8950 2023-12-07T23:46:20.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a02ffd640 10 monclient: dump: 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.133+0000 7f9a10d99640 4 set_mon_vals no callback set 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 10 monclient: _renew_subs 2023-12-07T23:46:20.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a141414d0 con 0x7f9a140a8950 2023-12-07T23:46:20.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 1 librados: init done 2023-12-07T23:46:20.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.134+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a140b72b0 con 0x7f9a140a8950 2023-12-07T23:46:20.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.151+0000 7f9a02ffd640 1 -- 172.21.15.129:0/3235718773 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9a0403b050 con 0x7f9a140a8950 2023-12-07T23:46:20.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.152+0000 7f9a02ffd640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f99f406c870 0x7f99f406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.152+0000 7f9a11d9b640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f99f406c870 0x7f99f406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.152+0000 7f9a11d9b640 10 monclient: get_auth_request con 0x7f99f406c870 auth_method 0 2023-12-07T23:46:20.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.153+0000 7f9a02ffd640 10 client.4390.objecter ms_handle_connect 0x7f99f406c870 2023-12-07T23:46:20.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.153+0000 7f9a11d9b640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f99f406c870 0x7f99f406ed30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a140b6f10 tx=0x7f9a08009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.155+0000 7f9a02ffd640 1 -- 172.21.15.129:0/3235718773 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9a0407b0c0 con 0x7f9a140a8950 2023-12-07T23:46:20.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.155+0000 7f9a02ffd640 10 client.4390.objecter ms_dispatch 0x7f9a14000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:20.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.155+0000 7f9a02ffd640 3 client.4390.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:20.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.155+0000 7f9a02ffd640 3 client.4390.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:20.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.155+0000 7f9a02ffd640 20 client.4390.objecter dump_active .. 0 homeless 2023-12-07T23:46:20.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-07T23:46:20.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-07T23:46:20.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.366+0000 7fe4dbdc9640 10 monclient: get_monmap_and_config 2023-12-07T23:46:20.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.366+0000 7fe4dbdc9640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.366+0000 7fe4dbdc9640 10 monclient: monmap: 2023-12-07T23:46:20.369 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.369 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.369 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:20.367303+0000 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:20.367303+0000 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.366+0000 7fe4dbdc9640 1 Processor -- start 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.366+0000 7fe4dbdc9640 1 -- start start 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.366+0000 7fe4dbdc9640 10 monclient: init 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe4d40a8970 0x7fe4d413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d40a8970 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4d40a9bd0 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4d413f3b0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40aa950 con 0x7fe4d413f3b0 2023-12-07T23:46:20.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40a90e0 con 0x7fe4d40a9bd0 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d40aabc0 con 0x7fe4d40a8970 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.367+0000 7fe4dbdc9640 10 monclient(hunting): authenticate will time out at 1380.555517s 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.368+0000 7fe4d9b65640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe4d40a8970 0x7fe4d413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.368+0000 7fe4da366640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fe4d413f3b0 0x7fe4d414c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43214/0 (socket says 172.21.15.129:43214) 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.368+0000 7fe4da366640 1 -- 172.21.15.129:0/3634645179 learned_addr learned my addr 172.21.15.129:0/3634645179 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.368+0000 7fe4d9b65640 10 monclient(hunting): get_auth_request con 0x7fe4d40a8970 auth_method 0 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.368+0000 7fe4d9b65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.368+0000 7fe4d9b65640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.368+0000 7fe4d9b65640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.369+0000 7fe4d9b65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.369+0000 7fe4d9b65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.369+0000 7fe4d9b65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 1 -- 172.21.15.129:0/3634645179 >> v1:172.21.15.184:6789/0 conn(0x7fe4d40a9bd0 legacy=0x7fe4d41421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 1 -- 172.21.15.129:0/3634645179 >> v1:172.21.15.129:6789/0 conn(0x7fe4d413f3b0 legacy=0x7fe4d414c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 1 monclient: found mon.noname-a 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:20.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.371+0000 7fe4d9b65640 1 -- 172.21.15.129:0/3634645179 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d40aaea0 con 0x7fe4d40a8970 2023-12-07T23:46:20.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.377+0000 7fe4d9b65640 1 --2- 172.21.15.129:0/3634645179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d40a8970 0x7fe4d413ee70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40097a0 tx=0x7fe4c402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8a31258b1b1527 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 1 -- 172.21.15.129:0/3634645179 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c402f050 con 0x7fe4d40a8970 2023-12-07T23:46:20.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 10 monclient: dump: 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 1 -- 172.21.15.129:0/3634645179 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c4031700 con 0x7fe4d40a8970 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 1 -- 172.21.15.129:0/3634645179 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4031a90 con 0x7fe4d40a8970 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4dbdc9640 5 monclient: authenticate success, global_id 4365 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4dbdc9640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4dbdc9640 4 set_mon_vals no callback set 2023-12-07T23:46:20.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.381+0000 7fe4d8b63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4d8b63640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4d8b63640 10 monclient: dump: 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.387 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 10 monclient: shutdown 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/3634645179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d40a8970 msgr2=0x7fe4d413ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/3634645179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d40a8970 0x7fe4d413ee70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40097a0 tx=0x7fe4c402b7a0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/3634645179 shutdown_connections 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/3634645179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d40a8970 0x7fe4d413ee70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/3634645179 >> 172.21.15.129:0/3634645179 conn(0x7fe4d4066dd0 msgr2=0x7fe4d40b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/3634645179 shutdown_connections 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.382+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/3634645179 wait complete. 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 10 monclient: monmap: 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.391 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 1 librados: starting msgr at 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 1 librados: starting objecter 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 1 Processor -- start 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 1 -- start start 2023-12-07T23:46:20.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 1 librados: setting wanted keys 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 1 librados: calling monclient init 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.383+0000 7fe4dbdc9640 10 monclient: init 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4d40a8970 0x7fe4d4143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d40a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4d40a9bd0 0x7fe4d413ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d40a9bd0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d413f3b0 0x7fe4d413d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d413f3b0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d4144b40 con 0x7fe4d413f3b0 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d41366e0 con 0x7fe4d40a9bd0 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d4145d00 con 0x7fe4d40a8970 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.384+0000 7fe4dbdc9640 10 monclient(hunting): authenticate will time out at 1380.572517s 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4da366640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d413f3b0 0x7fe4d413d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4d9b65640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4d40a8970 0x7fe4d4143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4da366640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d413f3b0 0x7fe4d413d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43118/0 (socket says 172.21.15.129:43118) 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4d9b65640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4d40a8970 0x7fe4d4143ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55784/0 (socket says 172.21.15.129:55784) 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4da366640 1 -- 172.21.15.129:0/2071129246 learned_addr learned my addr 172.21.15.129:0/2071129246 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4da366640 10 monclient(hunting): get_auth_request con 0x7fe4d413f3b0 auth_method 0 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4da366640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4da366640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4da366640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4d9b65640 10 monclient(hunting): get_auth_request con 0x7fe4d40a8970 auth_method 0 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4d9b65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4d9b65640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.385+0000 7fe4d9b65640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.386+0000 7fe4da366640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.386+0000 7fe4da366640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.386+0000 7fe4da366640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.386+0000 7fe4d9b65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.386+0000 7fe4d9b65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.386+0000 7fe4d9b65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.388+0000 7f9a02ffd640 1 -- 172.21.15.129:0/3235718773 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a04077400 con 0x7f9a140a8950 2023-12-07T23:46:20.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.388+0000 7f9a02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.388+0000 7f9a02ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.388+0000 7fe4da366640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 1 -- 172.21.15.129:0/2071129246 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4d40a8970 msgr2=0x7fe4d4143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4d40a8970 0x7fe4d4143ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 1 -- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4d40a9bd0 msgr2=0x7fe4d413ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4d40a9bd0 0x7fe4d413ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 1 monclient: found mon.noname-a 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.389+0000 7fe4da366640 1 -- 172.21.15.129:0/2071129246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c4009400 con 0x7fe4d413f3b0 2023-12-07T23:46:20.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.394+0000 7fe4da366640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d413f3b0 0x7fe4d413d380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00caa0 tx=0x7fe4cc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.394+0000 7fe4ca7fc640 10 client.?.objecter ms_handle_connect 0x7fe4d413f3b0 2023-12-07T23:46:20.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.394+0000 7fe4ca7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 1 -- 172.21.15.129:0/2071129246 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00ab40 con 0x7fe4d413f3b0 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: dump: 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 1 -- 172.21.15.129:0/2071129246 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc00aca0 con 0x7fe4d413f3b0 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 5 monclient: authenticate success, global_id 4371 2023-12-07T23:46:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 10 monclient: _renew_subs 2023-12-07T23:46:20.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:20.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d4145fe0 con 0x7fe4d413f3b0 2023-12-07T23:46:20.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 1 -- 172.21.15.129:0/2071129246 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc012600 con 0x7fe4d413f3b0 2023-12-07T23:46:20.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:20.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4ca7fc640 10 monclient: dump: 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4cbfff640 4 set_mon_vals no callback set 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 10 monclient: _renew_subs 2023-12-07T23:46:20.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:20.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d41445d0 con 0x7fe4d413f3b0 2023-12-07T23:46:20.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.398+0000 7fe4dbdc9640 1 librados: init done 2023-12-07T23:46:20.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.399+0000 7fe4dbdc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.399+0000 7fe4dbdc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.399+0000 7fe4dbdc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:20.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.399+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d4144cf0 con 0x7fe4d413f3b0 2023-12-07T23:46:20.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.419+0000 7fe4ca7fc640 1 -- 172.21.15.129:0/2071129246 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe4cc007070 con 0x7fe4d413f3b0 2023-12-07T23:46:20.422 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.419+0000 7fe4ca7fc640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe4b806c870 0x7fe4b806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.420+0000 7fe4d9b65640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe4b806c870 0x7fe4b806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.420+0000 7fe4d9b65640 10 monclient: get_auth_request con 0x7fe4b806c870 auth_method 0 2023-12-07T23:46:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.420+0000 7fe4ca7fc640 10 client.4371.objecter ms_handle_connect 0x7fe4b806c870 2023-12-07T23:46:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.420+0000 7fe4d9b65640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe4b806c870 0x7fe4b806ed30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4035000 tx=0x7fe4c4008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.427+0000 7f0514383640 10 monclient: get_monmap_and_config 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.427+0000 7f0514383640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.427+0000 7f0514383640 10 monclient: monmap: 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:20.428473+0000 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:20.428473+0000 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.430 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:20.431 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:20.431 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:20.431 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:20.431 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:20.431 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.427+0000 7f0514383640 1 Processor -- start 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.427+0000 7f0514383640 1 -- start start 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.427+0000 7f0514383640 10 monclient: init 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f050c0a8950 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c0b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient(hunting): picked mon.noname-b con 0x7f050c0a8d30 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f050c13f110 0x7f050c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient(hunting): picked mon.noname-a con 0x7f050c13f110 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:20.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c141e30 con 0x7f050c0a8950 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050c13fae0 con 0x7f050c13f110 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0a9150 con 0x7f050c0a8d30 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.428+0000 7f0514383640 10 monclient(hunting): authenticate will time out at 1380.616694s 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.429+0000 7f051211f640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f050c0a8950 0x7f050c1425a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:60112/0 (socket says 172.21.15.129:60112) 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.429+0000 7f051211f640 1 -- 172.21.15.129:0/3586233033 learned_addr learned my addr 172.21.15.129:0/3586233033 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.429+0000 7f051191e640 1 --2- 172.21.15.129:0/3586233033 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c0b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.429+0000 7f051191e640 10 monclient(hunting): get_auth_request con 0x7f050c0a8d30 auth_method 0 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.429+0000 7f051191e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.429+0000 7f051191e640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.429+0000 7f051191e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.430+0000 7f051191e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.430+0000 7f051191e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.435 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.430+0000 7f051191e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.431+0000 7f051111d640 1 -- 172.21.15.129:0/3586233033 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050012689 0 0) 0x7f050c141e30 con 0x7f050c0a8950 2023-12-07T23:46:20.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.431+0000 7f051111d640 10 monclient(hunting): _negotiate 2023-12-07T23:46:20.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.431+0000 7f051111d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.431+0000 7f051111d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.431+0000 7f051111d640 10 monclient(hunting): my global_id is 4393 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.431+0000 7f051111d640 1 -- 172.21.15.129:0/3586233033 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f8003520 con 0x7f050c0a8950 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 1 -- 172.21.15.129:0/3586233033 >> v2:172.21.15.129:3300/0 conn(0x7f050c13f110 msgr2=0x7f050c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 1 --2- 172.21.15.129:0/3586233033 >> v2:172.21.15.129:3300/0 conn(0x7f050c13f110 0x7f050c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 1 -- 172.21.15.129:0/3586233033 >> v1:172.21.15.184:6789/0 conn(0x7f050c0a8950 legacy=0x7f050c1425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 1 monclient: found mon.noname-b 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-07T23:46:20.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.432+0000 7f051191e640 1 -- 172.21.15.129:0/3586233033 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c0aa890 con 0x7f050c0a8d30 2023-12-07T23:46:20.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.434+0000 7f051191e640 1 --2- 172.21.15.129:0/3586233033 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c0b5610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0097a0 tx=0x7f04fc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=422eb3ff079ccf32 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 1 -- 172.21.15.129:0/3586233033 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc031830 con 0x7f050c0a8d30 2023-12-07T23:46:20.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: dump: 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 1 -- 172.21.15.129:0/3586233033 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc031990 con 0x7f050c0a8d30 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f0514383640 5 monclient: authenticate success, global_id 4304 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f0514383640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f0514383640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 1 -- 172.21.15.129:0/3586233033 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc031d20 con 0x7f050c0a8d30 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f0514383640 4 set_mon_vals no callback set 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:20.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.438+0000 7f051111d640 10 monclient: dump: 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.444 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 10 monclient: shutdown 2023-12-07T23:46:20.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:20.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 -- 172.21.15.129:0/3586233033 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 msgr2=0x7f050c0b5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 --2- 172.21.15.129:0/3586233033 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c0b5610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0097a0 tx=0x7f04fc02b8d0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 -- 172.21.15.129:0/3586233033 shutdown_connections 2023-12-07T23:46:20.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 --2- 172.21.15.129:0/3586233033 >> v2:172.21.15.129:3300/0 conn(0x7f050c13f110 0x7f050c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 --2- 172.21.15.129:0/3586233033 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c0b5610 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 -- 172.21.15.129:0/3586233033 >> 172.21.15.129:0/3586233033 conn(0x7f050c066cd0 msgr2=0x7f050c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 -- 172.21.15.129:0/3586233033 shutdown_connections 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.439+0000 7f0514383640 1 -- 172.21.15.129:0/3586233033 wait complete. 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 10 monclient: monmap: 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:20.448 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 1 librados: starting msgr at 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 1 librados: starting objecter 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 1 Processor -- start 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 1 -- start start 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 1 librados: setting wanted keys 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 1 librados: calling monclient init 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.440+0000 7f0514383640 10 monclient: init 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c0a8950 0x7f050c1458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): picked mon.noname-a con 0x7f050c0a8950 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:20.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): picked mon.noname-c con 0x7f050c0a8d30 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f050c13f110 0x7f050c143670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): picked mon.noname-b con 0x7f050c13f110 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c143bb0 con 0x7f050c0a8950 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c139090 con 0x7f050c13f110 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c139300 con 0x7f050c0a8d30 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0514383640 10 monclient(hunting): authenticate will time out at 1380.629425s 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f051191e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.441+0000 7f0512920640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f050c13f110 0x7f050c143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f050c13f110 0x7f050c143670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40888/0 (socket says 172.21.15.129:40888) 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 1 -- 172.21.15.129:0/2361416537 learned_addr learned my addr 172.21.15.129:0/2361416537 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 10 monclient(hunting): get_auth_request con 0x7f050c13f110 auth_method 0 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f051191e640 10 monclient(hunting): get_auth_request con 0x7f050c0a8d30 auth_method 0 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f051191e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f051191e640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f051191e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f0512920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f051191e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f051191e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.442+0000 7f051191e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 1 -- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f050c13f110 msgr2=0x7f050c143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f050c13f110 0x7f050c143670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 1 -- 172.21.15.129:0/2361416537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c0a8950 msgr2=0x7f050c1458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c0a8950 0x7f050c1458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 1 monclient: found mon.noname-c 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.445+0000 7f051191e640 1 -- 172.21.15.129:0/2361416537 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc009400 con 0x7f050c0a8d30 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.447+0000 7f051191e640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c143130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f050c0ab290 tx=0x7f04fc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.447+0000 7f0502ffd640 10 client.?.objecter ms_handle_connect 0x7f050c0a8d30 2023-12-07T23:46:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.447+0000 7f0502ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:20.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 1 -- 172.21.15.129:0/2361416537 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc03b9f0 con 0x7f050c0a8d30 2023-12-07T23:46:20.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:20.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: dump: 2023-12-07T23:46:20.455 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.455 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.455 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0514383640 5 monclient: authenticate success, global_id 4307 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 1 -- 172.21.15.129:0/2361416537 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc03bb90 con 0x7f050c0a8d30 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 1 -- 172.21.15.129:0/2361416537 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc0433f0 con 0x7f050c0a8d30 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0502ffd640 10 monclient: dump: 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f051091c640 4 set_mon_vals no callback set 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0514383640 10 monclient: _renew_subs 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0514383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.451+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c1395e0 con 0x7f050c0a8d30 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.452+0000 7f0514383640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.452+0000 7f0514383640 10 monclient: _renew_subs 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.452+0000 7f0514383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.452+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c145e40 con 0x7f050c0a8d30 2023-12-07T23:46:20.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.452+0000 7f0514383640 1 librados: init done 2023-12-07T23:46:20.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.453+0000 7f0514383640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.453+0000 7f0514383640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.453+0000 7f0514383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:20.459 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.453+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050c138140 con 0x7f050c0a8d30 2023-12-07T23:46:20.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-07T23:46:20.479 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.476+0000 7f0502ffd640 1 -- 172.21.15.129:0/2361416537 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f04fc043550 con 0x7f050c0a8d30 2023-12-07T23:46:20.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.477+0000 7f0502ffd640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f04f006c820 0x7f04f006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.477+0000 7f051211f640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f04f006c820 0x7f04f006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.480 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.477+0000 7f051211f640 10 monclient: get_auth_request con 0x7f04f006c820 auth_method 0 2023-12-07T23:46:20.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.478+0000 7f0502ffd640 10 client.4307.objecter ms_handle_connect 0x7f04f006c820 2023-12-07T23:46:20.481 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.478+0000 7f051211f640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f04f006c820 0x7f04f006ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0508000f60 tx=0x7f05080078f0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.480+0000 7f0502ffd640 1 -- 172.21.15.129:0/2361416537 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f04fc07a2d0 con 0x7f050c0a8d30 2023-12-07T23:46:20.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.480+0000 7f0502ffd640 10 client.4307.objecter ms_dispatch 0x7f050c000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:20.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.480+0000 7f0502ffd640 3 client.4307.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:20.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.480+0000 7f0502ffd640 3 client.4307.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:20.483 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.480+0000 7f0502ffd640 20 client.4307.objecter dump_active .. 0 homeless 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.561+0000 7feff1275640 10 monclient: get_monmap_and_config 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.561+0000 7feff1275640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.561+0000 7feff1275640 10 monclient: monmap: 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:20.563068+0000 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:20.563068+0000 2023-12-07T23:46:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.562+0000 7feff1275640 1 Processor -- start 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.562+0000 7feff1275640 1 -- start start 2023-12-07T23:46:20.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.562+0000 7feff1275640 10 monclient: init 2023-12-07T23:46:20.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a8970 0x7fefec13ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient(hunting): picked mon.noname-b con 0x7fefec0a8970 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefec0a9bd0 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fefec13f3b0 0x7fefec1428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient(hunting): picked mon.noname-c con 0x7fefec13f3b0 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec0aa950 con 0x7fefec0a9bd0 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefec142df0 con 0x7fefec13f3b0 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec13f840 con 0x7fefec0a8970 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7feff1275640 10 monclient(hunting): authenticate will time out at 1380.751225s 2023-12-07T23:46:20.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.563+0000 7fefeb7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fefec13f3b0 0x7fefec1428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fefec13f3b0 0x7fefec1428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40892/0 (socket says 172.21.15.129:40892) 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 1 -- 172.21.15.129:0/2738250434 learned_addr learned my addr 172.21.15.129:0/2738250434 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 10 monclient(hunting): get_auth_request con 0x7fefec13f3b0 auth_method 0 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.564+0000 7fefeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2023-12-07T23:46:20.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 1 -- 172.21.15.129:0/2738250434 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a8970 msgr2=0x7fefec13ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:20.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 1 --2- 172.21.15.129:0/2738250434 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a8970 0x7fefec13ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 1 -- 172.21.15.129:0/2738250434 >> v1:172.21.15.129:6789/0 conn(0x7fefec0a9bd0 legacy=0x7fefec1421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:20.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 1 monclient: found mon.noname-c 2023-12-07T23:46:20.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.566+0000 7fefeb7fe640 1 -- 172.21.15.129:0/2738250434 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec0a90e0 con 0x7fefec13f3b0 2023-12-07T23:46:20.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.568+0000 7fefeb7fe640 1 --2- 172.21.15.129:0/2738250434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec13f3b0 0x7fefec1428b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009e50 tx=0x7fefe402bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66ed7cd4c78884bf server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.570+0000 7f9a18894640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:20.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.570+0000 7f9a18894640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:20.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.570+0000 7f9a18894640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.570+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9a14138940 con 0x7f9a140a8950 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 1 -- 172.21.15.129:0/2738250434 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe402c950 con 0x7fefec13f3b0 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: dump: 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.575 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 5 monclient: authenticate success, global_id 4399 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 1 -- 172.21.15.129:0/2738250434 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe402cab0 con 0x7fefec13f3b0 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 4 set_mon_vals no callback set 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 1 -- 172.21.15.129:0/2738250434 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe402ce40 con 0x7fefec13f3b0 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7fefe9ffb640 10 monclient: dump: 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.576 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.577 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 10 monclient: shutdown 2023-12-07T23:46:20.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:20.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 -- 172.21.15.129:0/2738250434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec13f3b0 msgr2=0x7fefec1428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 --2- 172.21.15.129:0/2738250434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec13f3b0 0x7fefec1428b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009e50 tx=0x7fefe402bdb0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 -- 172.21.15.129:0/2738250434 shutdown_connections 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 --2- 172.21.15.129:0/2738250434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec13f3b0 0x7fefec1428b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 --2- 172.21.15.129:0/2738250434 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a8970 0x7fefec13ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 -- 172.21.15.129:0/2738250434 >> 172.21.15.129:0/2738250434 conn(0x7fefec066dd0 msgr2=0x7fefec0b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 -- 172.21.15.129:0/2738250434 shutdown_connections 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.572+0000 7feff1275640 1 -- 172.21.15.129:0/2738250434 wait complete. 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 10 monclient: build_initial_monmap 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 10 monclient: monmap: 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.580 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 1 librados: starting msgr at 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 1 librados: starting objecter 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 1 Processor -- start 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 1 -- start start 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 1 librados: setting wanted keys 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 1 librados: calling monclient init 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.573+0000 7feff1275640 10 monclient: init 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec0a8970 0x7fefec13e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): picked mon.noname-b con 0x7fefec0a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a9bd0 0x7fefec133a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): picked mon.noname-c con 0x7fefec0a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec13f3b0 0x7fefec131c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): picked mon.noname-a con 0x7fefec13f3b0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec135e60 con 0x7fefec13f3b0 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec136760 con 0x7fefec0a8970 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec136940 con 0x7fefec0a9bd0 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.574+0000 7feff1275640 10 monclient(hunting): authenticate will time out at 1380.762400s 2023-12-07T23:46:20.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.575+0000 7f9a02ffd640 1 -- 172.21.15.129:0/3235718773 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a14138940 con 0x7f9a140a8950 2023-12-07T23:46:20.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.575+0000 7f9a02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T23:46:20.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.575+0000 7f9a02ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:20.583 INFO:teuthology.orchestra.run.smithi129.stdout:55834574853 2023-12-07T23:46:20.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 10 librados: watch_flush enter 2023-12-07T23:46:20.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 10 librados: watch_flush exit 2023-12-07T23:46:20.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 20 client.4390.objecter shutdown clearing up homeless session... 2023-12-07T23:46:20.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 10 client.4390.objecter successfully canceled tick 2023-12-07T23:46:20.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f99f406c870 msgr2=0x7f99f406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f99f406c870 0x7f99f406ed30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a140b6f10 tx=0x7f9a08009040 comp rx=0 tx=0).stop 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 10 monclient: shutdown 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 msgr2=0x7f9a1413a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a1413a350 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009d90 tx=0x7f9a04004a40 comp rx=0 tx=0).stop 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 shutdown_connections 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f99f406c870 0x7f99f406ed30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1413f110 0x7f9a141453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a140b5610 0x7f9a14144b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 --2- 172.21.15.129:0/3235718773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a140a8950 0x7f9a1413a350 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.577+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 >> 172.21.15.129:0/3235718773 conn(0x7f9a14066cd0 msgr2=0x7f9a14073f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.578+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 shutdown_connections 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.578+0000 7f9a18894640 1 -- 172.21.15.129:0/3235718773 wait complete. 2023-12-07T23:46:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.578+0000 7f9a18894640 1 librados: shutdown 2023-12-07T23:46:20.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.579+0000 7fefeaffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec0a8970 0x7fefec13e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.580+0000 7fefeaffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec0a8970 0x7fefec13e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:40904/0 (socket says 172.21.15.129:40904) 2023-12-07T23:46:20.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.580+0000 7fefeaffd640 1 -- 172.21.15.129:0/357434912 learned_addr learned my addr 172.21.15.129:0/357434912 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:20.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.580+0000 7fefeaffd640 10 monclient(hunting): get_auth_request con 0x7fefec0a8970 auth_method 0 2023-12-07T23:46:20.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.580+0000 7fefeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.580+0000 7fefeaffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.580+0000 7fefeaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.581+0000 7fefeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.581+0000 7fefeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.581+0000 7fefeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 1 -- 172.21.15.129:0/357434912 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a9bd0 msgr2=0x7fefec133a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a9bd0 0x7fefec133a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 1 -- 172.21.15.129:0/357434912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec13f3b0 msgr2=0x7fefec131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec13f3b0 0x7fefec131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 10 monclient: _finish_hunting 0 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 1 monclient: found mon.noname-b 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.584+0000 7fefeaffd640 1 -- 172.21.15.129:0/357434912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4009b30 con 0x7fefec0a8970 2023-12-07T23:46:20.589 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.586+0000 7fefeaffd640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec0a8970 0x7fefec13e2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00b4d0 tx=0x7fefdc00b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.589 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.586+0000 7fefd37fe640 10 client.?.objecter ms_handle_connect 0x7fefec0a8970 2023-12-07T23:46:20.589 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.586+0000 7fefd37fe640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:20.592 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2023-12-07T23:46:20.592 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 1 -- 172.21.15.129:0/357434912 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc010870 con 0x7fefec0a8970 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: dump: 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.593 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 1 -- 172.21.15.129:0/357434912 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc0109d0 con 0x7fefec0a8970 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 1 -- 172.21.15.129:0/357434912 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc010d80 con 0x7fefec0a8970 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7feff1275640 5 monclient: authenticate success, global_id 4402 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7feff1275640 10 monclient: _renew_subs 2023-12-07T23:46:20.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7feff1275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec136060 con 0x7fefec0a8970 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefe97fa640 4 set_mon_vals no callback set 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7fefd37fe640 10 monclient: dump: 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:20.595 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:20.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7feff1275640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:20.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.589+0000 7feff1275640 10 monclient: _renew_subs 2023-12-07T23:46:20.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.590+0000 7feff1275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.590+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec134d00 con 0x7fefec0a8970 2023-12-07T23:46:20.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.590+0000 7feff1275640 1 librados: init done 2023-12-07T23:46:20.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.591+0000 7feff1275640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.591+0000 7feff1275640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.591+0000 7feff1275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:20.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.591+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec134f80 con 0x7fefec0a8970 2023-12-07T23:46:20.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.608+0000 7fefd37fe640 1 -- 172.21.15.129:0/357434912 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fefdc023480 con 0x7fefec0a8970 2023-12-07T23:46:20.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.609+0000 7fefd37fe640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fefd806c850 0x7fefd806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:20.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.609+0000 7fefea7fc640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fefd806c850 0x7fefd806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:20.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.609+0000 7fefea7fc640 10 monclient: get_auth_request con 0x7fefd806c850 auth_method 0 2023-12-07T23:46:20.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.609+0000 7fefea7fc640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fefd806c850 0x7fefd806ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fefec1359d0 tx=0x7fefe0007990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:20.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.609+0000 7fefd37fe640 10 client.4402.objecter ms_handle_connect 0x7fefd806c850 2023-12-07T23:46:20.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.611+0000 7fefd37fe640 1 -- 172.21.15.129:0/357434912 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fefdc058e70 con 0x7fefec0a8970 2023-12-07T23:46:20.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.611+0000 7fefd37fe640 10 client.4402.objecter ms_dispatch 0x7fefec000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:20.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.611+0000 7fefd37fe640 3 client.4402.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:20.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.611+0000 7fefd37fe640 3 client.4402.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:20.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.611+0000 7fefd37fe640 20 client.4402.objecter dump_active .. 0 homeless 2023-12-07T23:46:20.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 1 -- 172.21.15.129:0/2071129246 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe4cc058540 con 0x7fe4d413f3b0 2023-12-07T23:46:20.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 10 client.4371.objecter ms_dispatch 0x7fe4d4000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:20.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 3 client.4371.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:20.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 3 client.4371.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:20.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 20 client.4371.objecter dump_active .. 0 homeless 2023-12-07T23:46:20.688 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 1 -- 172.21.15.129:0/2071129246 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4cc054880 con 0x7fe4d413f3b0 2023-12-07T23:46:20.689 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.689 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.685+0000 7fe4ca7fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:20.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.733+0000 7f0502ffd640 1 -- 172.21.15.129:0/2361416537 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04fc076610 con 0x7f050c0a8d30 2023-12-07T23:46:20.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.734+0000 7f0502ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.737 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.734+0000 7f0502ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:20.843 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.840+0000 7fefd37fe640 1 -- 172.21.15.129:0/357434912 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefdc054ab0 con 0x7fefec0a8970 2023-12-07T23:46:20.843 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.840+0000 7fefd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:20.843 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.840+0000 7fefd37fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:20.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.865+0000 7fe4dbdc9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:20.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.865+0000 7fe4dbdc9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:20.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.865+0000 7fe4dbdc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-07T23:46:20.869 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.865+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe4d404e840 con 0x7fe4d413f3b0 2023-12-07T23:46:20.913 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.910+0000 7fe4ca7fc640 1 -- 172.21.15.129:0/2071129246 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4cc017090 con 0x7fe4d413f3b0 2023-12-07T23:46:20.913 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.910+0000 7fe4ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T23:46:20.913 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.910+0000 7fe4ca7fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:20.913 INFO:teuthology.orchestra.run.smithi129.stdout:68719476742 2023-12-07T23:46:20.915 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 10 librados: watch_flush enter 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 10 librados: watch_flush exit 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 20 client.4371.objecter shutdown clearing up homeless session... 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 10 client.4371.objecter successfully canceled tick 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe4b806c870 msgr2=0x7fe4b806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe4b806c870 0x7fe4b806ed30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4035000 tx=0x7fe4c4008030 comp rx=0 tx=0).stop 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 10 monclient: shutdown 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d413f3b0 msgr2=0x7fe4d413d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d413f3b0 0x7fe4d413d380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00caa0 tx=0x7fe4cc00cf60 comp rx=0 tx=0).stop 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 shutdown_connections 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fe4b806c870 0x7fe4b806ed30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4d413f3b0 0x7fe4d413d380 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4d40a9bd0 0x7fe4d413ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 --2- 172.21.15.129:0/2071129246 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe4d40a8970 0x7fe4d4143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.917 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 >> 172.21.15.129:0/2071129246 conn(0x7fe4d4066dd0 msgr2=0x7fe4d40b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:20.917 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 shutdown_connections 2023-12-07T23:46:20.917 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 -- 172.21.15.129:0/2071129246 wait complete. 2023-12-07T23:46:20.917 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.912+0000 7fe4dbdc9640 1 librados: shutdown 2023-12-07T23:46:20.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.916+0000 7f0514383640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:20.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.916+0000 7f0514383640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:20.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.916+0000 7f0514383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:46:20.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.916+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f050c139440 con 0x7f050c0a8d30 2023-12-07T23:46:20.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.3 2023-12-07T23:46:20.928 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:20.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.956+0000 7f0502ffd640 1 -- 172.21.15.129:0/2361416537 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f050c139440 con 0x7f050c0a8d30 2023-12-07T23:46:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.956+0000 7f0502ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T23:46:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.956+0000 7f0502ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:20.960 INFO:teuthology.orchestra.run.smithi129.stdout:68719476742 2023-12-07T23:46:20.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 10 librados: watch_flush enter 2023-12-07T23:46:20.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 10 librados: watch_flush exit 2023-12-07T23:46:20.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 20 client.4307.objecter shutdown clearing up homeless session... 2023-12-07T23:46:20.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 10 client.4307.objecter successfully canceled tick 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f04f006c820 msgr2=0x7f04f006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f04f006c820 0x7f04f006ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0508000f60 tx=0x7f05080078f0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 10 monclient: shutdown 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 msgr2=0x7f050c143130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c143130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f050c0ab290 tx=0x7f04fc0046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 shutdown_connections 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f04f006c820 0x7f04f006ece0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f050c13f110 0x7f050c143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f050c0a8d30 0x7f050c143130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 --2- 172.21.15.129:0/2361416537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c0a8950 0x7f050c1458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.959+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 >> 172.21.15.129:0/2361416537 conn(0x7f050c066cd0 msgr2=0x7f050c13c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:20.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.960+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 shutdown_connections 2023-12-07T23:46:20.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.960+0000 7f0514383640 1 -- 172.21.15.129:0/2361416537 wait complete. 2023-12-07T23:46:20.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:20.960+0000 7f0514383640 1 librados: shutdown 2023-12-07T23:46:20.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.2 2023-12-07T23:46:20.975 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:21.030 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.027+0000 7feff1275640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:21.031 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.027+0000 7feff1275640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:21.031 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.027+0000 7feff1275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.031 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.027+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fefec04e840 con 0x7fefec0a8970 2023-12-07T23:46:21.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.033+0000 7fefd37fe640 1 -- 172.21.15.129:0/357434912 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fefdc01e020 con 0x7fefec0a8970 2023-12-07T23:46:21.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.033+0000 7fefd37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T23:46:21.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.033+0000 7fefd37fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:21.036 INFO:teuthology.orchestra.run.smithi129.stdout:55834574853 2023-12-07T23:46:21.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 10 librados: watch_flush enter 2023-12-07T23:46:21.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 10 librados: watch_flush exit 2023-12-07T23:46:21.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 20 client.4402.objecter shutdown clearing up homeless session... 2023-12-07T23:46:21.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 10 client.4402.objecter successfully canceled tick 2023-12-07T23:46:21.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fefd806c850 msgr2=0x7fefd806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fefd806c850 0x7fefd806ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fefec1359d0 tx=0x7fefe0007990 comp rx=0 tx=0).stop 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 10 monclient: shutdown 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec0a8970 msgr2=0x7fefec13e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.035+0000 7feff1275640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec0a8970 0x7fefec13e2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00b4d0 tx=0x7fefdc00b990 comp rx=0 tx=0).stop 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 shutdown_connections 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fefd806c850 0x7fefd806ed10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec13f3b0 0x7fefec131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fefec0a9bd0 0x7fefec133a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 --2- 172.21.15.129:0/357434912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefec0a8970 0x7fefec13e2c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 >> 172.21.15.129:0/357434912 conn(0x7fefec066dd0 msgr2=0x7fefec130920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 shutdown_connections 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 -- 172.21.15.129:0/357434912 wait complete. 2023-12-07T23:46:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.036+0000 7feff1275640 1 librados: shutdown 2023-12-07T23:46:21.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.4 2023-12-07T23:46:21.050 DEBUG:teuthology.parallel:result is None 2023-12-07T23:46:21.050 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-07T23:46:21.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T23:46:21.150 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.147+0000 7f1ae3aec640 10 monclient: get_monmap_and_config 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.147+0000 7f1ae3aec640 10 monclient: build_initial_monmap 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.147+0000 7f1ae3aec640 10 monclient: monmap: 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:21.148670+0000 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:21.148670+0000 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:21.151 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:21.152 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 1 Processor -- start 2023-12-07T23:46:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 1 -- start start 2023-12-07T23:46:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 10 monclient: init 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1adc0af180 0x7f1adc13d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 10 monclient(hunting): picked mon.noname-a con 0x7f1adc0af180 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1adc13dd90 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1adc142000 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.148+0000 7f1ae3aec640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1adc0a9810 con 0x7f1adc142000 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae3aec640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1adc0abb60 con 0x7f1adc13dd90 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae3aec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae3aec640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1adc142430 con 0x7f1adc0af180 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae3aec640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae3aec640 10 monclient(hunting): authenticate will time out at 1381.337004s 2023-12-07T23:46:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae1888640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1adc0af180 0x7f1adc13d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae1087640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f1adc13dd90 0x7f1adc1418f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:60116/0 (socket says 172.21.15.129:60116) 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.149+0000 7f1ae1087640 1 -- 172.21.15.129:0/3331566795 learned_addr learned my addr 172.21.15.129:0/3331566795 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.150+0000 7f1ae1888640 10 monclient(hunting): get_auth_request con 0x7f1adc0af180 auth_method 0 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.150+0000 7f1ae1888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.150+0000 7f1ae1888640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.150+0000 7f1ae1888640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.150+0000 7f1ae1888640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.150+0000 7f1ae1888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.150+0000 7f1ae1888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae0886640 1 -- 172.21.15.129:0/3331566795 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326724040 0 0) 0x7f1adc0abb60 con 0x7f1adc13dd90 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae0886640 10 monclient(hunting): _negotiate 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae0886640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae0886640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae0886640 10 monclient(hunting): my global_id is 4405 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae0886640 1 -- 172.21.15.129:0/3331566795 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac8003520 con 0x7f1adc13dd90 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 1 -- 172.21.15.129:0/3331566795 >> v1:172.21.15.184:6789/0 conn(0x7f1adc13dd90 legacy=0x7f1adc1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 1 -- 172.21.15.129:0/3331566795 >> v1:172.21.15.129:6789/0 conn(0x7f1adc142000 legacy=0x7f1adc148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 10 monclient: _finish_hunting 0 2023-12-07T23:46:21.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 1 monclient: found mon.noname-a 2023-12-07T23:46:21.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:21.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:21.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.152+0000 7f1ae1888640 1 -- 172.21.15.129:0/3331566795 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc13e1b0 con 0x7f1adc0af180 2023-12-07T23:46:21.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.155+0000 7f1ae1888640 1 --2- 172.21.15.129:0/3331566795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc0af180 0x7f1adc13d850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800a940 tx=0x7f1ad802cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578f452b9fd0a17c server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 1 -- 172.21.15.129:0/3331566795 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad802db40 con 0x7f1adc0af180 2023-12-07T23:46:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: dump: 2023-12-07T23:46:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: _finish_auth 0 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: _check_auth_tickets 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 1 -- 172.21.15.129:0/3331566795 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad802dca0 con 0x7f1adc0af180 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 1 -- 172.21.15.129:0/3331566795 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8035660 con 0x7f1adc0af180 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae0886640 10 monclient: dump: 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 5 monclient: authenticate success, global_id 4374 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:21.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 4 set_mon_vals no callback set 2023-12-07T23:46:21.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 10 monclient: shutdown 2023-12-07T23:46:21.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:21.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 1 -- 172.21.15.129:0/3331566795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc0af180 msgr2=0x7f1adc13d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/3331566795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc0af180 0x7f1adc13d850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800a940 tx=0x7f1ad802cd90 comp rx=0 tx=0).stop 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 1 -- 172.21.15.129:0/3331566795 shutdown_connections 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/3331566795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc0af180 0x7f1adc13d850 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 1 -- 172.21.15.129:0/3331566795 >> 172.21.15.129:0/3331566795 conn(0x7f1adc06ac00 msgr2=0x7f1adc0b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.161+0000 7f1ae3aec640 1 -- 172.21.15.129:0/3331566795 shutdown_connections 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.162+0000 7f1ae3aec640 1 -- 172.21.15.129:0/3331566795 wait complete. 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.162+0000 7f1ae3aec640 10 monclient: build_initial_monmap 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.162+0000 7f1ae3aec640 10 monclient: monmap: 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:21.170 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.162+0000 7f1ae3aec640 1 librados: starting msgr at 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.162+0000 7f1ae3aec640 1 librados: starting objecter 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 Processor -- start 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 -- start start 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 librados: setting wanted keys 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 librados: calling monclient init 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient: init 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:46:21.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1adc0af180 0x7f1adc140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): picked mon.noname-b con 0x7f1adc0af180 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1adc13dd90 0x7f1adc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): picked mon.noname-c con 0x7f1adc13dd90 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc142000 0x7f1adc13b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): picked mon.noname-a con 0x7f1adc142000 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc06e8c0 con 0x7f1adc142000 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc138e90 con 0x7f1adc0af180 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc140790 con 0x7f1adc13dd90 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.163+0000 7f1ae3aec640 10 monclient(hunting): authenticate will time out at 1381.351655s 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae2089640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc142000 0x7f1adc13b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1087640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1adc13dd90 0x7f1adc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1888640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1adc0af180 0x7f1adc140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae2089640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc142000 0x7f1adc13b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43178/0 (socket says 172.21.15.129:43178) 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae2089640 1 -- 172.21.15.129:0/4065243418 learned_addr learned my addr 172.21.15.129:0/4065243418 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae2089640 10 monclient(hunting): get_auth_request con 0x7f1adc142000 auth_method 0 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae2089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae2089640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae2089640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1888640 10 monclient(hunting): get_auth_request con 0x7f1adc0af180 auth_method 0 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1888640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1888640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1087640 10 monclient(hunting): get_auth_request con 0x7f1adc13dd90 auth_method 0 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1087640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1087640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.164+0000 7f1ae1087640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae1888640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae1888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae1888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae2089640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae2089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae2089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae1087640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae1087640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.165+0000 7f1ae1087640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 1 -- 172.21.15.129:0/4065243418 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1adc13dd90 msgr2=0x7f1adc06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1adc13dd90 0x7f1adc06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 1 -- 172.21.15.129:0/4065243418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc142000 msgr2=0x7f1adc13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc142000 0x7f1adc13b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 10 monclient: _finish_hunting 0 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 1 monclient: found mon.noname-b 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.167+0000 7f1ae1888640 1 -- 172.21.15.129:0/4065243418 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad800a5a0 con 0x7f1adc0af180 2023-12-07T23:46:21.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.169+0000 7f1ae1888640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1adc0af180 0x7f1adc140250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8034000 tx=0x7f1ad80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:21.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.169+0000 7f1ad27fc640 10 client.?.objecter ms_handle_connect 0x7f1adc0af180 2023-12-07T23:46:21.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.169+0000 7f1ad27fc640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:21.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 1 -- 172.21.15.129:0/4065243418 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad803e610 con 0x7f1adc0af180 2023-12-07T23:46:21.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: dump: 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: _finish_auth 0 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: _check_auth_tickets 2023-12-07T23:46:21.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 1 -- 172.21.15.129:0/4065243418 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad803e770 con 0x7f1adc0af180 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 1 -- 172.21.15.129:0/4065243418 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad803eb20 con 0x7f1adc0af180 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 5 monclient: authenticate success, global_id 4408 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 10 monclient: _renew_subs 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1adc06eaf0 con 0x7f1adc0af180 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad3fff640 4 set_mon_vals no callback set 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ad27fc640 10 monclient: dump: 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.178 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 10 monclient: _renew_subs 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc1cb530 con 0x7f1adc0af180 2023-12-07T23:46:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.173+0000 7f1ae3aec640 1 librados: init done 2023-12-07T23:46:21.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.174+0000 7f1ae3aec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:21.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.174+0000 7f1ae3aec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:21.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.174+0000 7f1ae3aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.180 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.174+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1adc04e840 con 0x7f1adc0af180 2023-12-07T23:46:21.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.417+0000 7f1ad27fc640 1 -- 172.21.15.129:0/4065243418 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1ad803c040 con 0x7f1adc0af180 2023-12-07T23:46:21.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.417+0000 7f1ad27fc640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1ac006c870 0x7f1ac006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.417+0000 7f1ae1087640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1ac006c870 0x7f1ac006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.417+0000 7f1ae1087640 10 monclient: get_auth_request con 0x7f1ac006c870 auth_method 0 2023-12-07T23:46:21.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.418+0000 7f1ae1087640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1ac006c870 0x7f1ac006ed30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1adc141800 tx=0x7f1acc007d30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:21.421 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.418+0000 7f1ad27fc640 10 client.4408.objecter ms_handle_connect 0x7f1ac006c870 2023-12-07T23:46:21.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.420+0000 7f1ad27fc640 1 -- 172.21.15.129:0/4065243418 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1ad807aa60 con 0x7f1adc0af180 2023-12-07T23:46:21.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.420+0000 7f1ad27fc640 10 client.4408.objecter ms_dispatch 0x7f1adc000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:21.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.420+0000 7f1ad27fc640 3 client.4408.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:21.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.420+0000 7f1ad27fc640 3 client.4408.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:21.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.420+0000 7f1ad27fc640 20 client.4408.objecter dump_active .. 0 homeless 2023-12-07T23:46:21.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.429+0000 7f1ad27fc640 1 -- 172.21.15.129:0/4065243418 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ad8076da0 con 0x7f1adc0af180 2023-12-07T23:46:21.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.429+0000 7f1ad27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:21.432 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.429+0000 7f1ad27fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:21.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.601+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1adc0ac0b0 con 0x7f1ac006c870 2023-12-07T23:46:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.605+0000 7f1ad27fc640 1 -- 172.21.15.129:0/4065243418 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47974 (secure 0 0 0) 0x7f1acc005850 con 0x7f1ac006c870 2023-12-07T23:46:21.608 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:46:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-07T23:46:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 10 librados: watch_flush enter 2023-12-07T23:46:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 10 librados: watch_flush exit 2023-12-07T23:46:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 20 client.4408.objecter shutdown clearing up homeless session... 2023-12-07T23:46:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 10 client.4408.objecter successfully canceled tick 2023-12-07T23:46:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1ac006c870 msgr2=0x7f1ac006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1ac006c870 0x7f1ac006ed30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1adc141800 tx=0x7f1acc007d30 comp rx=0 tx=0).stop 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 10 monclient: shutdown 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1adc0af180 msgr2=0x7f1adc140250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.606+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1adc0af180 0x7f1adc140250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8034000 tx=0x7f1ad80046c0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 shutdown_connections 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f1ac006c870 0x7f1ac006ed30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1adc142000 0x7f1adc13b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1adc13dd90 0x7f1adc06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 --2- 172.21.15.129:0/4065243418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1adc0af180 0x7f1adc140250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 >> 172.21.15.129:0/4065243418 conn(0x7f1adc06ac00 msgr2=0x7f1adc073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 shutdown_connections 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 -- 172.21.15.129:0/4065243418 wait complete. 2023-12-07T23:46:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.607+0000 7f1ae3aec640 1 librados: shutdown 2023-12-07T23:46:21.620 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-12-07T23:46:21.547809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218716,"kb_used_data":2360,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756004,"statfs":{"total":773094113280,"available":772870148096,"internally_reserved":0,"allocated":2416640,"data_stored":1359218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001525"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.071466+0000","last_change":"2023-12-07T23:46:13.620414+0000","last_active":"2023-12-07T23:46:16.071466+0000","last_peered":"2023-12-07T23:46:16.071466+0000","last_clean":"2023-12-07T23:46:16.071466+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-07T23:46:16.071466+0000","last_undegraded":"2023-12-07T23:46:16.071466+0000","last_fullsized":"2023-12-07T23:46:16.071466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:44:19.783064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:15.861603+0000","last_change":"2023-12-07T23:46:13.656681+0000","last_active":"2023-12-07T23:46:15.861603+0000","last_peered":"2023-12-07T23:46:15.861603+0000","last_clean":"2023-12-07T23:46:15.861603+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-07T23:46:15.861603+0000","last_undegraded":"2023-12-07T23:46:15.861603+0000","last_fullsized":"2023-12-07T23:46:15.861603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:29.359895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.063949+0000","last_change":"2023-12-07T23:46:13.447293+0000","last_active":"2023-12-07T23:46:16.063949+0000","last_peered":"2023-12-07T23:46:16.063949+0000","last_clean":"2023-12-07T23:46:16.063949+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-07T23:46:16.063949+0000","last_undegraded":"2023-12-07T23:46:16.063949+0000","last_fullsized":"2023-12-07T23:46:16.063949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:00.005960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:15.821591+0000","last_change":"2023-12-07T23:46:13.905749+0000","last_active":"2023-12-07T23:46:15.821591+0000","last_peered":"2023-12-07T23:46:15.821591+0000","last_clean":"2023-12-07T23:46:15.821591+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-07T23:46:15.821591+0000","last_undegraded":"2023-12-07T23:46:15.821591+0000","last_fullsized":"2023-12-07T23:46:15.821591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:45:13.859049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.047723+0000","last_change":"2023-12-07T23:46:13.873757+0000","last_active":"2023-12-07T23:46:16.047723+0000","last_peered":"2023-12-07T23:46:16.047723+0000","last_clean":"2023-12-07T23:46:16.047723+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-07T23:46:16.047723+0000","last_undegraded":"2023-12-07T23:46:16.047723+0000","last_fullsized":"2023-12-07T23:46:16.047723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:40:06.824492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.037873+0000","last_change":"2023-12-07T23:46:13.343801+0000","last_active":"2023-12-07T23:46:16.037873+0000","last_peered":"2023-12-07T23:46:16.037873+0000","last_clean":"2023-12-07T23:46:16.037873+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-07T23:46:16.037873+0000","last_undegraded":"2023-12-07T23:46:16.037873+0000","last_fullsized":"2023-12-07T23:46:16.037873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:42:20.201085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.034380+0000","last_change":"2023-12-07T23:46:13.418137+0000","last_active":"2023-12-07T23:46:16.034380+0000","last_peered":"2023-12-07T23:46:16.034380+0000","last_clean":"2023-12-07T23:46:16.034380+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-07T23:46:16.034380+0000","last_undegraded":"2023-12-07T23:46:16.034380+0000","last_fullsized":"2023-12-07T23:46:16.034380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:51:13.255892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.053580+0000","last_change":"2023-12-07T23:46:13.815701+0000","last_active":"2023-12-07T23:46:16.053580+0000","last_peered":"2023-12-07T23:46:16.053580+0000","last_clean":"2023-12-07T23:46:16.053580+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-07T23:46:16.053580+0000","last_undegraded":"2023-12-07T23:46:16.053580+0000","last_fullsized":"2023-12-07T23:46:16.053580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:13.731531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":118,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-07T23:46:13.732713+0000","last_change":"2023-12-07T23:46:05.583767+0000","last_active":"2023-12-07T23:46:13.732713+0000","last_peered":"2023-12-07T23:46:13.732713+0000","last_clean":"2023-12-07T23:46:13.732713+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-07T23:46:13.732713+0000","last_undegraded":"2023-12-07T23:46:13.732713+0000","last_fullsized":"2023-12-07T23:46:13.732713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_clean_scrub_stamp":"2023-12-07T23:46:02.581722+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:48:03.670610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476743,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.155999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.774000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.114000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":48.348999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":33.593000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":47.281999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.436}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.010999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.649000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.094999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.318000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.901}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.763000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.384}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.945}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.973000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3640000000000008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6269999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.481}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":469677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1389999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.726000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.396000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":165.20099999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":138.44900000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.815}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8919999999999995}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3520000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.984}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.741}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.943000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.721}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.98}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.151}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":60679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":152.363}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7240000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.353000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.016999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.335000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.239999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.895}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":606208,"data_stored":469677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7409999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.369999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.606}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.108000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.646000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.555}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.574000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.866}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":215.363}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.664}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":33.460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":246.09700000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.414999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T23:46:21.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.726+0000 7f9c4eba4640 10 monclient: get_monmap_and_config 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.726+0000 7f9c4eba4640 10 monclient: build_initial_monmap 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.726+0000 7f9c4eba4640 10 monclient: monmap: 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:21.727488+0000 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:21.727488+0000 2023-12-07T23:46:21.729 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:21.730 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.730 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:21.730 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:21.730 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:21.730 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:21.730 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:21.730 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 1 Processor -- start 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 1 -- start start 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient: init 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9c480af180 0x7f9c4813d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c480af180 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c4813dd90 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9c48142000 0x7f9c48148c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.732 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient(hunting): picked mon.noname-c con 0x7f9c48142000 addr v2:172.21.15.184:3300/0 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480a9810 con 0x7f9c4813dd90 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c48142430 con 0x7f9c480af180 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480af560 con 0x7f9c48142000 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.727+0000 7f9c4eba4640 10 monclient(hunting): authenticate will time out at 1381.915706s 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4c940640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9c480af180 0x7f9c4813d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4d141640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9c48142000 0x7f9c48148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4c940640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9c480af180 0x7f9c4813d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43194/0 (socket says 172.21.15.129:43194) 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4c940640 1 -- 172.21.15.129:0/1230113932 learned_addr learned my addr 172.21.15.129:0/1230113932 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4c940640 10 monclient(hunting): get_auth_request con 0x7f9c480af180 auth_method 0 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4c940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4c940640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.733 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.728+0000 7f9c4c940640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4d141640 10 monclient(hunting): get_auth_request con 0x7f9c48142000 auth_method 0 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4d141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4d141640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4d141640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4c940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4c940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4c940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4d141640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4d141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.734 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.729+0000 7f9c4d141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 1 -- 172.21.15.129:0/1230113932 >> v2:172.21.15.184:3300/0 conn(0x7f9c48142000 msgr2=0x7f9c48148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 1 --2- 172.21.15.129:0/1230113932 >> v2:172.21.15.184:3300/0 conn(0x7f9c48142000 0x7f9c48148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 1 -- 172.21.15.129:0/1230113932 >> v1:172.21.15.184:6789/0 conn(0x7f9c4813dd90 legacy=0x7f9c481418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 10 monclient: _finish_hunting 0 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 1 monclient: found mon.noname-a 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:21.735 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.730+0000 7f9c4c940640 1 -- 172.21.15.129:0/1230113932 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480abb60 con 0x7f9c480af180 2023-12-07T23:46:21.736 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.733+0000 7f9c4c940640 1 --2- 172.21.15.129:0/1230113932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 0x7f9c4813d850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9c4813f490 tx=0x7f9c3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dd4569147c1db7 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:21.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 1 -- 172.21.15.129:0/1230113932 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30031870 con 0x7f9c480af180 2023-12-07T23:46:21.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:21.740 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: dump: 2023-12-07T23:46:21.740 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.740 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: _finish_auth 0 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: _check_auth_tickets 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 1 -- 172.21.15.129:0/1230113932 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c300319d0 con 0x7f9c480af180 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 1 -- 172.21.15.129:0/1230113932 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30031d60 con 0x7f9c480af180 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.741 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c477fe640 10 monclient: dump: 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c4eba4640 5 monclient: authenticate success, global_id 4383 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c4eba4640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.736+0000 7f9c4eba4640 4 set_mon_vals no callback set 2023-12-07T23:46:21.742 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 10 monclient: shutdown 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 -- 172.21.15.129:0/1230113932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 msgr2=0x7f9c4813d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/1230113932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 0x7f9c4813d850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9c4813f490 tx=0x7f9c3002b910 comp rx=0 tx=0).stop 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 -- 172.21.15.129:0/1230113932 shutdown_connections 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/1230113932 >> v2:172.21.15.184:3300/0 conn(0x7f9c48142000 0x7f9c48148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/1230113932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 0x7f9c4813d850 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 -- 172.21.15.129:0/1230113932 >> 172.21.15.129:0/1230113932 conn(0x7f9c4806ac00 msgr2=0x7f9c480b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 -- 172.21.15.129:0/1230113932 shutdown_connections 2023-12-07T23:46:21.743 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.737+0000 7f9c4eba4640 1 -- 172.21.15.129:0/1230113932 wait complete. 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 10 monclient: build_initial_monmap 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 10 monclient: monmap: 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.746 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 1 librados: starting msgr at 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 1 librados: starting objecter 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 1 Processor -- start 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 1 -- start start 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 1 librados: setting wanted keys 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 1 librados: calling monclient init 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.738+0000 7f9c4eba4640 10 monclient: init 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 0x7f9c4814b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c480af180 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:21.747 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c4813dd90 0x7f9c48130290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): picked mon.noname-c con 0x7f9c4813dd90 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c48142000 0x7f9c4812e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): picked mon.noname-b con 0x7f9c48142000 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4814cee0 con 0x7f9c480af180 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4814d330 con 0x7f9c48142000 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4813d320 con 0x7f9c4813dd90 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.739+0000 7f9c4eba4640 10 monclient(hunting): authenticate will time out at 1381.927531s 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c4813dd90 0x7f9c48130290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c4d141640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c48142000 0x7f9c4812e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c4813dd90 0x7f9c48130290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55830/0 (socket says 172.21.15.129:55830) 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c47fff640 1 -- 172.21.15.129:0/745143939 learned_addr learned my addr 172.21.15.129:0/745143939 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:21.748 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c47fff640 10 monclient(hunting): get_auth_request con 0x7f9c4813dd90 auth_method 0 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c47fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.740+0000 7f9c47fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4d141640 10 monclient(hunting): get_auth_request con 0x7f9c48142000 auth_method 0 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4d141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4d141640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4d141640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4d141640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4d141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4d141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.741+0000 7f9c4c940640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 0x7f9c4814b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.742+0000 7f9c4c940640 10 monclient(hunting): get_auth_request con 0x7f9c480af180 auth_method 0 2023-12-07T23:46:21.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.742+0000 7f9c4c940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:21.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.742+0000 7f9c4c940640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:21.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.742+0000 7f9c4c940640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:21.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.742+0000 7f9c4c940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:21.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.742+0000 7f9c4c940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:21.750 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.742+0000 7f9c4c940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:21.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2023-12-07T23:46:21.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 1 -- 172.21.15.129:0/745143939 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c4813dd90 msgr2=0x7f9c48130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c4813dd90 0x7f9c48130290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 1 -- 172.21.15.129:0/745143939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 msgr2=0x7f9c4814b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:21.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 0x7f9c4814b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:21.751 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 10 monclient: _finish_hunting 0 2023-12-07T23:46:21.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 1 monclient: found mon.noname-b 2023-12-07T23:46:21.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:21.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.744+0000 7f9c4d141640 1 -- 172.21.15.129:0/745143939 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300094d0 con 0x7f9c48142000 2023-12-07T23:46:21.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.746+0000 7f9c4d141640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c48142000 0x7f9c4812e560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000ebd0 tx=0x7f9c4000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:21.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.746+0000 7f9c457fa640 10 client.?.objecter ms_handle_connect 0x7f9c48142000 2023-12-07T23:46:21.752 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.746+0000 7f9c457fa640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 1 -- 172.21.15.129:0/745143939 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4000ccc0 con 0x7f9c48142000 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: dump: 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.753 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: _finish_auth 0 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: _check_auth_tickets 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 1 -- 172.21.15.129:0/745143939 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4000f040 con 0x7f9c48142000 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 1 -- 172.21.15.129:0/745143939 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40014480 con 0x7f9c48142000 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 5 monclient: authenticate success, global_id 4414 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 10 monclient: _renew_subs 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4813e1f0 con 0x7f9c48142000 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c46ffd640 4 set_mon_vals no callback set 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c457fa640 10 monclient: dump: 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 10 monclient: _renew_subs 2023-12-07T23:46:21.755 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c481307d0 con 0x7f9c48142000 2023-12-07T23:46:21.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.750+0000 7f9c4eba4640 1 librados: init done 2023-12-07T23:46:21.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.751+0000 7f9c4eba4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:21.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.751+0000 7f9c4eba4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:21.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.751+0000 7f9c4eba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:21.756 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.751+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4814d050 con 0x7f9c48142000 2023-12-07T23:46:21.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.993+0000 7f9c457fa640 1 -- 172.21.15.129:0/745143939 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9c400145e0 con 0x7f9c48142000 2023-12-07T23:46:21.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.994+0000 7f9c457fa640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9c2806c850 0x7f9c2806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:21.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.994+0000 7f9c4c940640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9c2806c850 0x7f9c2806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:21.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.994+0000 7f9c4c940640 10 monclient: get_auth_request con 0x7f9c2806c850 auth_method 0 2023-12-07T23:46:21.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.994+0000 7f9c4c940640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9c2806c850 0x7f9c2806ed10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9c30036000 tx=0x7f9c3003c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:21.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.994+0000 7f9c457fa640 10 client.4414.objecter ms_handle_connect 0x7f9c2806c850 2023-12-07T23:46:22.000 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.997+0000 7f9c457fa640 1 -- 172.21.15.129:0/745143939 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9c40059390 con 0x7f9c48142000 2023-12-07T23:46:22.000 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.997+0000 7f9c457fa640 10 client.4414.objecter ms_dispatch 0x7f9c48000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:22.000 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.997+0000 7f9c457fa640 3 client.4414.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:22.000 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.997+0000 7f9c457fa640 3 client.4414.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:22.000 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:21.997+0000 7f9c457fa640 20 client.4414.objecter dump_active .. 0 homeless 2023-12-07T23:46:22.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.008+0000 7f9c457fa640 1 -- 172.21.15.129:0/745143939 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c400556d0 con 0x7f9c48142000 2023-12-07T23:46:22.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.008+0000 7f9c457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:22.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.008+0000 7f9c457fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.204+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c480ac0b0 con 0x7f9c2806c850 2023-12-07T23:46:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.206+0000 7f9c457fa640 1 -- 172.21.15.129:0/745143939 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47974 (secure 0 0 0) 0x7f9c30044400 con 0x7f9c2806c850 2023-12-07T23:46:22.210 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:46:22.210 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 10 librados: watch_flush enter 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 10 librados: watch_flush exit 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 20 client.4414.objecter shutdown clearing up homeless session... 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 10 client.4414.objecter successfully canceled tick 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9c2806c850 msgr2=0x7f9c2806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9c2806c850 0x7f9c2806ed10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9c30036000 tx=0x7f9c3003c040 comp rx=0 tx=0).stop 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 10 monclient: shutdown 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c48142000 msgr2=0x7f9c4812e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c48142000 0x7f9c4812e560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000ebd0 tx=0x7f9c4000c490 comp rx=0 tx=0).stop 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 shutdown_connections 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f9c2806c850 0x7f9c2806ed10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c48142000 0x7f9c4812e560 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c4813dd90 0x7f9c48130290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 --2- 172.21.15.129:0/745143939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c480af180 0x7f9c4814b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 >> 172.21.15.129:0/745143939 conn(0x7f9c4806ac00 msgr2=0x7f9c48073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 shutdown_connections 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 -- 172.21.15.129:0/745143939 wait complete. 2023-12-07T23:46:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.208+0000 7f9c4eba4640 1 librados: shutdown 2023-12-07T23:46:22.223 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-12-07T23:46:21.547809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218716,"kb_used_data":2360,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756004,"statfs":{"total":773094113280,"available":772870148096,"internally_reserved":0,"allocated":2416640,"data_stored":1359218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001525"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.071466+0000","last_change":"2023-12-07T23:46:13.620414+0000","last_active":"2023-12-07T23:46:16.071466+0000","last_peered":"2023-12-07T23:46:16.071466+0000","last_clean":"2023-12-07T23:46:16.071466+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-07T23:46:16.071466+0000","last_undegraded":"2023-12-07T23:46:16.071466+0000","last_fullsized":"2023-12-07T23:46:16.071466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:44:19.783064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:15.861603+0000","last_change":"2023-12-07T23:46:13.656681+0000","last_active":"2023-12-07T23:46:15.861603+0000","last_peered":"2023-12-07T23:46:15.861603+0000","last_clean":"2023-12-07T23:46:15.861603+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-07T23:46:15.861603+0000","last_undegraded":"2023-12-07T23:46:15.861603+0000","last_fullsized":"2023-12-07T23:46:15.861603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:29.359895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.063949+0000","last_change":"2023-12-07T23:46:13.447293+0000","last_active":"2023-12-07T23:46:16.063949+0000","last_peered":"2023-12-07T23:46:16.063949+0000","last_clean":"2023-12-07T23:46:16.063949+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-07T23:46:16.063949+0000","last_undegraded":"2023-12-07T23:46:16.063949+0000","last_fullsized":"2023-12-07T23:46:16.063949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:00.005960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:15.821591+0000","last_change":"2023-12-07T23:46:13.905749+0000","last_active":"2023-12-07T23:46:15.821591+0000","last_peered":"2023-12-07T23:46:15.821591+0000","last_clean":"2023-12-07T23:46:15.821591+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-07T23:46:15.821591+0000","last_undegraded":"2023-12-07T23:46:15.821591+0000","last_fullsized":"2023-12-07T23:46:15.821591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:45:13.859049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.047723+0000","last_change":"2023-12-07T23:46:13.873757+0000","last_active":"2023-12-07T23:46:16.047723+0000","last_peered":"2023-12-07T23:46:16.047723+0000","last_clean":"2023-12-07T23:46:16.047723+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-07T23:46:16.047723+0000","last_undegraded":"2023-12-07T23:46:16.047723+0000","last_fullsized":"2023-12-07T23:46:16.047723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:40:06.824492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.037873+0000","last_change":"2023-12-07T23:46:13.343801+0000","last_active":"2023-12-07T23:46:16.037873+0000","last_peered":"2023-12-07T23:46:16.037873+0000","last_clean":"2023-12-07T23:46:16.037873+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-07T23:46:16.037873+0000","last_undegraded":"2023-12-07T23:46:16.037873+0000","last_fullsized":"2023-12-07T23:46:16.037873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:42:20.201085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.034380+0000","last_change":"2023-12-07T23:46:13.418137+0000","last_active":"2023-12-07T23:46:16.034380+0000","last_peered":"2023-12-07T23:46:16.034380+0000","last_clean":"2023-12-07T23:46:16.034380+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-07T23:46:16.034380+0000","last_undegraded":"2023-12-07T23:46:16.034380+0000","last_fullsized":"2023-12-07T23:46:16.034380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:51:13.255892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-07T23:46:16.053580+0000","last_change":"2023-12-07T23:46:13.815701+0000","last_active":"2023-12-07T23:46:16.053580+0000","last_peered":"2023-12-07T23:46:16.053580+0000","last_clean":"2023-12-07T23:46:16.053580+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-07T23:46:16.053580+0000","last_undegraded":"2023-12-07T23:46:16.053580+0000","last_fullsized":"2023-12-07T23:46:16.053580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:13.731531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":118,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-07T23:46:13.732713+0000","last_change":"2023-12-07T23:46:05.583767+0000","last_active":"2023-12-07T23:46:13.732713+0000","last_peered":"2023-12-07T23:46:13.732713+0000","last_clean":"2023-12-07T23:46:13.732713+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-07T23:46:13.732713+0000","last_undegraded":"2023-12-07T23:46:13.732713+0000","last_fullsized":"2023-12-07T23:46:13.732713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_clean_scrub_stamp":"2023-12-07T23:46:02.581722+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:48:03.670610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476743,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.155999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.774000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.114000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":48.348999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":33.593000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":47.281999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.436}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.010999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.649000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.094999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.318000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.901}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.763000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.384}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.945}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.973000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3640000000000008}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6269999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.481}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":469677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1389999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.726000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.396000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":165.20099999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":138.44900000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.815}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8919999999999995}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3520000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.984}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.741}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.943000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.721}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.98}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.151}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":60679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":152.363}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7240000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.353000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.016999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.335000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.239999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.895}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":606208,"data_stored":469677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7409999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.369999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.606}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.108000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.646000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.555}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.574000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.866}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":215.363}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.664}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":33.460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":246.09700000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.414999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T23:46:22.224 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-07T23:46:22.224 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-07T23:46:22.224 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-07T23:46:22.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-07T23:46:22.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.319+0000 7fa3f790f640 10 monclient: get_monmap_and_config 2023-12-07T23:46:22.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.319+0000 7fa3f790f640 10 monclient: build_initial_monmap 2023-12-07T23:46:22.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.319+0000 7fa3f790f640 10 monclient: monmap: 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:46:22.320469+0000 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:46:22.320469+0000 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.319+0000 7fa3f790f640 1 Processor -- start 2023-12-07T23:46:22.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 1 -- start start 2023-12-07T23:46:22.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient: init 2023-12-07T23:46:22.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:22.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-12-07T23:46:22.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3f006c570 addr v1:172.21.15.129:6789/0 2023-12-07T23:46:22.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa3f013f7d0 0x7fa3f006c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:22.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f013f7d0 addr v2:172.21.15.129:3300/0 2023-12-07T23:46:22.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3f01422b0 addr v1:172.21.15.184:6789/0 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f0141e20 con 0x7fa3f006c570 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f006ce90 con 0x7fa3f01422b0 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f006bb70 con 0x7fa3f013f7d0 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.320+0000 7fa3f790f640 10 monclient(hunting): authenticate will time out at 1382.508664s 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.321+0000 7fa3f4eaa640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa3f013f7d0 0x7fa3f006c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.321+0000 7fa3f5eac640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fa3f01422b0 0x7fa3f0142740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:60126/0 (socket says 172.21.15.129:60126) 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.321+0000 7fa3f5eac640 1 -- 172.21.15.129:0/3614885462 learned_addr learned my addr 172.21.15.129:0/3614885462 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.321+0000 7fa3f4eaa640 10 monclient(hunting): get_auth_request con 0x7fa3f013f7d0 auth_method 0 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.321+0000 7fa3f4eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.321+0000 7fa3f4eaa640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.321+0000 7fa3f4eaa640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:22.326 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.322+0000 7fa3f4eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:22.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.322+0000 7fa3f4eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:22.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.322+0000 7fa3f4eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:22.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.323+0000 7fa3e7fff640 1 -- 172.21.15.129:0/3614885462 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255734029 0 0) 0x7fa3f006ce90 con 0x7fa3f01422b0 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.323+0000 7fa3e7fff640 10 monclient(hunting): _negotiate 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.323+0000 7fa3e7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.323+0000 7fa3e7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.323+0000 7fa3e7fff640 10 monclient(hunting): my global_id is 4417 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.323+0000 7fa3e7fff640 1 -- 172.21.15.129:0/3614885462 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc003520 con 0x7fa3f01422b0 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 10 monclient(hunting): handle_auth_done global_id 4389 payload 274 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 1 -- 172.21.15.129:0/3614885462 >> v1:172.21.15.184:6789/0 conn(0x7fa3f01422b0 legacy=0x7fa3f0142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 1 -- 172.21.15.129:0/3614885462 >> v1:172.21.15.129:6789/0 conn(0x7fa3f006c570 legacy=0x7fa3f013f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 10 monclient: _finish_hunting 0 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 1 monclient: found mon.noname-a 2023-12-07T23:46:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:22.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-07T23:46:22.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.324+0000 7fa3f4eaa640 1 -- 172.21.15.129:0/3614885462 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00709a0 con 0x7fa3f013f7d0 2023-12-07T23:46:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.329+0000 7fa3f4eaa640 1 --2- 172.21.15.129:0/3614885462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f006c950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00097a0 tx=0x7fa3e002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8822f7083e86d77f server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 1 -- 172.21.15.129:0/3614885462 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0030610 con 0x7fa3f013f7d0 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: dump: 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:22.336 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: _finish_auth 0 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: _check_auth_tickets 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 1 -- 172.21.15.129:0/3614885462 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e0030770 con 0x7fa3f013f7d0 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 1 -- 172.21.15.129:0/3614885462 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0030b00 con 0x7fa3f013f7d0 2023-12-07T23:46:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 5 monclient: authenticate success, global_id 4389 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 10 monclient: get_monmap_and_config success 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 4 set_mon_vals no callback set 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3e7fff640 10 monclient: dump: 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.339 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:22.339 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:22.339 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:22.339 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:22.339 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:22.339 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 10 monclient: shutdown 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 1 -- 172.21.15.129:0/3614885462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 msgr2=0x7fa3f006c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 1 --2- 172.21.15.129:0/3614885462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f006c950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00097a0 tx=0x7fa3e002b960 comp rx=0 tx=0).stop 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 1 -- 172.21.15.129:0/3614885462 shutdown_connections 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 1 --2- 172.21.15.129:0/3614885462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f006c950 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 1 -- 172.21.15.129:0/3614885462 >> 172.21.15.129:0/3614885462 conn(0x7fa3f0066b90 msgr2=0x7fa3f00b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:22.343 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 1 -- 172.21.15.129:0/3614885462 shutdown_connections 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.333+0000 7fa3f790f640 1 -- 172.21.15.129:0/3614885462 wait complete. 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.334+0000 7fa3f790f640 10 monclient: build_initial_monmap 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.334+0000 7fa3f790f640 10 monclient: monmap: 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.334+0000 7fa3f790f640 1 librados: starting msgr at 2023-12-07T23:46:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.334+0000 7fa3f790f640 1 librados: starting objecter 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 Processor -- start 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 -- start start 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 librados: setting wanted keys 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 librados: calling monclient init 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient: init 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient: _reopen_session rank -1 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa3f006c570 0x7fa3f0134360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f006c570 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f0131640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f013f7d0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3f01422b0 0x7fa3f0131b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f01422b0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0135840 con 0x7fa3f013f7d0 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0133ed0 con 0x7fa3f01422b0 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): start opening mon connection 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0143500 con 0x7fa3f006c570 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): _renew_subs 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.335+0000 7fa3f790f640 10 monclient(hunting): authenticate will time out at 1382.523577s 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f0131640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3f01422b0 0x7fa3f0131b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f56ab640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa3f006c570 0x7fa3f0134360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f0131640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43234/0 (socket says 172.21.15.129:43234) 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 1 -- 172.21.15.129:0/1361884808 learned_addr learned my addr 172.21.15.129:0/1361884808 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 10 monclient(hunting): get_auth_request con 0x7fa3f013f7d0 auth_method 0 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 10 monclient(hunting): get_auth_request con 0x7fa3f01422b0 auth_method 0 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f56ab640 10 monclient(hunting): get_auth_request con 0x7fa3f006c570 auth_method 0 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f56ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f56ab640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f56ab640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f4eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.336+0000 7fa3f5eac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.337+0000 7fa3f56ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.337+0000 7fa3f56ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.337+0000 7fa3f56ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 1 -- 172.21.15.129:0/1361884808 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa3f006c570 msgr2=0x7fa3f0134360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa3f006c570 0x7fa3f0134360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 1 -- 172.21.15.129:0/1361884808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 msgr2=0x7fa3f0131640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f0131640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 10 monclient: _finish_hunting 0 2023-12-07T23:46:22.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 1 monclient: found mon.noname-b 2023-12-07T23:46:22.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:46:22.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-07T23:46:22.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.339+0000 7fa3f5eac640 1 -- 172.21.15.129:0/1361884808 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0009400 con 0x7fa3f01422b0 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.341+0000 7fa3f5eac640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3f01422b0 0x7fa3f0131b80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800caa0 tx=0x7fa3e800cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.341+0000 7fa3e5ffb640 10 client.?.objecter ms_handle_connect 0x7fa3f01422b0 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.341+0000 7fa3e5ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 1 -- 172.21.15.129:0/1361884808 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800ab40 con 0x7fa3f01422b0 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: dump: 2023-12-07T23:46:22.351 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: _finish_auth 0 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: _check_auth_tickets 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 1 -- 172.21.15.129:0/1361884808 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e800aca0 con 0x7fa3f01422b0 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:46:22.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 1 -- 172.21.15.129:0/1361884808 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8012600 con 0x7fa3f01422b0 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e5ffb640 10 monclient: dump: 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:46:22.353 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 5 monclient: authenticate success, global_id 4420 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 10 monclient: _renew_subs 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3e77fe640 4 set_mon_vals no callback set 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f0135aa0 con 0x7fa3f01422b0 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 10 monclient: _renew_subs 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f01431a0 con 0x7fa3f01422b0 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.345+0000 7fa3f790f640 1 librados: init done 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.346+0000 7fa3f790f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.346+0000 7fa3f790f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:22.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.346+0000 7fa3f790f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:22.355 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.346+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f0134080 con 0x7fa3f01422b0 2023-12-07T23:46:22.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.364+0000 7fa3e5ffb640 1 -- 172.21.15.129:0/1361884808 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa3e8007070 con 0x7fa3f01422b0 2023-12-07T23:46:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.364+0000 7fa3e5ffb640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fa3d406c870 0x7fa3d406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:46:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.364+0000 7fa3f56ab640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fa3d406c870 0x7fa3d406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:46:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.364+0000 7fa3f56ab640 10 monclient: get_auth_request con 0x7fa3d406c870 auth_method 0 2023-12-07T23:46:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.365+0000 7fa3e5ffb640 10 client.4420.objecter ms_handle_connect 0x7fa3d406c870 2023-12-07T23:46:22.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.365+0000 7fa3f56ab640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fa3d406c870 0x7fa3d406ed30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec001780 tx=0x7fa3ec009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:46:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.367+0000 7fa3e5ffb640 1 -- 172.21.15.129:0/1361884808 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa3e8058790 con 0x7fa3f01422b0 2023-12-07T23:46:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.367+0000 7fa3e5ffb640 10 client.4420.objecter ms_dispatch 0x7fa3f0000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T23:46:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.367+0000 7fa3e5ffb640 3 client.4420.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:46:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.367+0000 7fa3e5ffb640 3 client.4420.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:46:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.367+0000 7fa3e5ffb640 20 client.4420.objecter dump_active .. 0 homeless 2023-12-07T23:46:22.598 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.595+0000 7fa3e5ffb640 1 -- 172.21.15.129:0/1361884808 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3e8054ad0 con 0x7fa3f01422b0 2023-12-07T23:46:22.599 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.595+0000 7fa3e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T23:46:22.599 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.595+0000 7fa3e5ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T23:46:22.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.806+0000 7fa3f790f640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-12-07T23:46:22.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.806+0000 7fa3f790f640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-12-07T23:46:22.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.806+0000 7fa3f790f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-07T23:46:22.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.806+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa3f004e840 con 0x7fa3f01422b0 2023-12-07T23:46:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.812+0000 7fa3e5ffb640 1 -- 172.21.15.129:0/1361884808 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa3e8021080 con 0x7fa3f01422b0 2023-12-07T23:46:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.812+0000 7fa3e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-12-07T23:46:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.812+0000 7fa3e5ffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T23:46:22.816 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-07T23:46:22.816 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-07T23:46:22.817 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.813+0000 7fa3f790f640 10 librados: watch_flush enter 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 10 librados: watch_flush exit 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 20 client.4420.objecter shutdown clearing up homeless session... 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 10 client.4420.objecter successfully canceled tick 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fa3d406c870 msgr2=0x7fa3d406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fa3d406c870 0x7fa3d406ed30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec001780 tx=0x7fa3ec009040 comp rx=0 tx=0).stop 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 10 monclient: shutdown 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:46:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3f01422b0 msgr2=0x7fa3f0131b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3f01422b0 0x7fa3f0131b80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800caa0 tx=0x7fa3e800cea0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 shutdown_connections 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fa3d406c870 0x7fa3d406ed30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3f01422b0 0x7fa3f0131b80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f013f7d0 0x7fa3f0131640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 --2- 172.21.15.129:0/1361884808 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa3f006c570 0x7fa3f0134360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 >> 172.21.15.129:0/1361884808 conn(0x7fa3f0066b90 msgr2=0x7fa3f0073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:46:22.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 shutdown_connections 2023-12-07T23:46:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 -- 172.21.15.129:0/1361884808 wait complete. 2023-12-07T23:46:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-07T23:46:22.814+0000 7fa3f790f640 1 librados: shutdown 2023-12-07T23:46:22.828 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-07T23:46:22.829 INFO:teuthology.run_tasks:Running task workunit... 2023-12-07T23:46:22.845 INFO:tasks.workunit:Pulling workunits from ref c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-07T23:46:22.846 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-07T23:46:22.846 DEBUG:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-07T23:46:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T23:46:22.861 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-07T23:46:22.861 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-07T23:46:22.917 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-07T23:46:22.917 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-07T23:46:22.972 INFO:tasks.workunit:timeout=6h 2023-12-07T23:46:22.972 INFO:tasks.workunit:cleanup=True 2023-12-07T23:46:22.973 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c56ad3eaf0784383bf22c04e35d26e15c4baeafe 2023-12-07T23:46:23.027 INFO:tasks.workunit.client.0.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-07T23:46:33.602 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:30.075 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.606 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:30.079 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.654 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:30.099 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.655 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:30.128 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.661 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:30.106 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.664 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:30.137 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.677 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:30.149 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.713 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:30.158 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:33.717 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:30.190 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.619 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:50.091 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.701 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:50.174 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.728 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:01:50.201 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.750 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:50.195 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.752 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:50.197 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.766 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:01:50.211 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.778 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:50.251 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.780 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:50.253 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:46:53.799 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:01:50.271 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:08.903 INFO:tasks.workunit.client.0.smithi129.stderr:Note: switching to 'c56ad3eaf0784383bf22c04e35d26e15c4baeafe'. 2023-12-07T23:47:08.903 INFO:tasks.workunit.client.0.smithi129.stderr: 2023-12-07T23:47:08.903 INFO:tasks.workunit.client.0.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-07T23:47:08.904 INFO:tasks.workunit.client.0.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-07T23:47:08.904 INFO:tasks.workunit.client.0.smithi129.stderr:state without impacting any branches by switching back to a branch. 2023-12-07T23:47:08.904 INFO:tasks.workunit.client.0.smithi129.stderr: 2023-12-07T23:47:08.904 INFO:tasks.workunit.client.0.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-07T23:47:08.904 INFO:tasks.workunit.client.0.smithi129.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-07T23:47:08.904 INFO:tasks.workunit.client.0.smithi129.stderr: 2023-12-07T23:47:08.904 INFO:tasks.workunit.client.0.smithi129.stderr: git switch -c 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr: 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr:Or undo this operation with: 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr: 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr: git switch - 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr: 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr: 2023-12-07T23:47:08.905 INFO:tasks.workunit.client.0.smithi129.stderr:HEAD is now at c56ad3eaf07 Merge branch 'wip-neorados-learning-from-experience' of https://github.com/adamemerson/ceph into wip-yuri3-testing-2023-12-07-0727 2023-12-07T23:47:08.908 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-07T23:47:08.971 INFO:tasks.workunit.client.0.smithi129.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-07T23:47:08.973 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-07T23:47:08.973 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-07T23:47:09.662 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-07T23:47:09.736 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-07T23:47:09.777 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-07T23:47:09.779 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-07T23:47:09.779 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-07T23:47:09.830 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-07T23:47:09.834 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-12-07T23:47:09.834 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-07T23:47:09.889 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-12-07T23:47:09.890 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-12-07T23:47:09.891 DEBUG:teuthology.orchestra.run.smithi129:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c56ad3eaf0784383bf22c04e35d26e15c4baeafe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-12-07T23:47:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:+ parallel=1 2023-12-07T23:47:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' '' = --serial ']' 2023-12-07T23:47:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:+ color= 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -t 1 ']' 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:+ declare -A pids 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_aio 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_aio' 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_aio 2023-12-07T23:47:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:09.951 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_aio 2023-12-07T23:47:09.951 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92381 2023-12-07T23:47:09.951 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_aio on pid 92381' 2023-12-07T23:47:09.952 INFO:tasks.workunit.client.0.smithi129.stdout:test api_aio on pid 92381 2023-12-07T23:47:09.955 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92381 2023-12-07T23:47:09.955 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:09.955 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:09.955 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-12-07T23:47:09.955 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_aio_pp 2023-12-07T23:47:09.955 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_aio_pp' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_aio_pp 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:09.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_aio_pp 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92388 2023-12-07T23:47:09.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_aio_pp on pid 92388' 2023-12-07T23:47:09.958 INFO:tasks.workunit.client.0.smithi129.stdout:test api_aio_pp on pid 92388 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92388 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_io 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_io' 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_io 2023-12-07T23:47:09.961 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:09.963 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.964 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:09.965 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.965 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:09.965 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_io 2023-12-07T23:47:09.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92397 2023-12-07T23:47:09.965 INFO:tasks.workunit.client.0.smithi129.stdout:test api_io on pid 92397 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_io on pid 92397' 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92397 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_io_pp 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_io_pp' 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_io_pp 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:09.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:09.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:09.972 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_io_pp 2023-12-07T23:47:09.973 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92409 2023-12-07T23:47:09.974 INFO:tasks.workunit.client.0.smithi129.stdout:test api_io_pp on pid 92409 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_io_pp on pid 92409' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92409 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_asio 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_asio' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_asio 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:09.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_asio 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92423 2023-12-07T23:47:09.984 INFO:tasks.workunit.client.0.smithi129.stdout:test api_asio on pid 92423 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_asio on pid 92423' 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92423 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:09.991 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_list 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_list' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_list 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:09.992 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:09.993 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:09.994 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:09.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:09.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92381/exe 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_list 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92442 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_list on pid 92442' 2023-12-07T23:47:09.998 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-12-07T23:47:09.999 INFO:tasks.workunit.client.0.smithi129.stdout:test api_list on pid 92442 2023-12-07T23:47:10.006 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92442 2023-12-07T23:47:10.006 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.006 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.006 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_lock 2023-12-07T23:47:10.006 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_lock' 2023-12-07T23:47:10.006 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_lock 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.007 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.008 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_aio.log 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_aio: /' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92388/exe 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.013 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_lock 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92460 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_lock on pid 92460' 2023-12-07T23:47:10.014 INFO:tasks.workunit.client.0.smithi129.stdout:test api_lock on pid 92460 2023-12-07T23:47:10.022 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92460 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_lock_pp 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_lock_pp' 2023-12-07T23:47:10.023 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_lock_pp 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.027 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.028 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_aio_pp: /' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.029 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92397/exe 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_lock_pp 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92479 2023-12-07T23:47:10.030 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_lock_pp on pid 92479' 2023-12-07T23:47:10.031 INFO:tasks.workunit.client.0.smithi129.stdout:test api_lock_pp on pid 92479 2023-12-07T23:47:10.037 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-12-07T23:47:10.037 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92479 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_misc 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_misc' 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_misc 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.038 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.039 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.042 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92409/exe 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_io.log 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_misc 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92496 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_misc on pid 92496' 2023-12-07T23:47:10.046 INFO:tasks.workunit.client.0.smithi129.stdout:test api_misc on pid 92496 2023-12-07T23:47:10.053 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92496 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_io: /' 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_misc_pp 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_misc_pp' 2023-12-07T23:47:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_io_pp: /' 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_misc_pp 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_misc_pp 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.060 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.060 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.060 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92514 2023-12-07T23:47:10.060 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.060 INFO:tasks.workunit.client.0.smithi129.stdout:test api_misc_pp on pid 92514 2023-12-07T23:47:10.070 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_misc_pp on pid 92514' 2023-12-07T23:47:10.070 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92514 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_tier_pp 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_tier_pp' 2023-12-07T23:47:10.073 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_tier_pp 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.077 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92442/exe 2023-12-07T23:47:10.078 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_tier_pp 2023-12-07T23:47:10.079 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.080 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.080 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92532 2023-12-07T23:47:10.080 INFO:tasks.workunit.client.0.smithi129.stdout:test api_tier_pp on pid 92532 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_tier_pp on pid 92532' 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92532 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.086 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92423/exe 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_pool 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_pool' 2023-12-07T23:47:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_pool 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_list: /' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.088 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.089 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_pool 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92552 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_pool on pid 92552' 2023-12-07T23:47:10.090 INFO:tasks.workunit.client.0.smithi129.stdout:test api_pool on pid 92552 2023-12-07T23:47:10.107 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92552 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_list.log 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_asio.log 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.108 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.109 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_asio: /' 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_snapshots 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_snapshots' 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.116 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92460/exe 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_snapshots 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92479/exe 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_lock.log 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_lock: /' 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_lock_pp: /' 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_snapshots 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92595 2023-12-07T23:47:10.124 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_snapshots on pid 92595' 2023-12-07T23:47:10.125 INFO:tasks.workunit.client.0.smithi129.stdout:test api_snapshots on pid 92595 2023-12-07T23:47:10.130 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92595 2023-12-07T23:47:10.130 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.130 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.130 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_snapshots_pp 2023-12-07T23:47:10.130 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_snapshots_pp' 2023-12-07T23:47:10.130 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.133 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.135 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_snapshots_pp 2023-12-07T23:47:10.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_snapshots_pp 2023-12-07T23:47:10.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92611 2023-12-07T23:47:10.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_snapshots_pp on pid 92611' 2023-12-07T23:47:10.135 INFO:tasks.workunit.client.0.smithi129.stdout:test api_snapshots_pp on pid 92611 2023-12-07T23:47:10.145 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92611 2023-12-07T23:47:10.145 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.145 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.145 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-12-07T23:47:10.145 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2023-12-07T23:47:10.145 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.146 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.147 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_stat 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_stat' 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92514/exe 2023-12-07T23:47:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_stat 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.153 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_stat 2023-12-07T23:47:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.155 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.155 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.155 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92648 2023-12-07T23:47:10.155 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.155 INFO:tasks.workunit.client.0.smithi129.stdout:test api_stat on pid 92648 2023-12-07T23:47:10.165 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_stat on pid 92648' 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92648 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.166 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.168 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.169 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.170 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_misc_pp: /' 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_stat_pp 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_stat_pp' 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_stat_pp 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.171 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.174 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92496/exe 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_stat_pp 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92682 2023-12-07T23:47:10.175 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_stat_pp on pid 92682' 2023-12-07T23:47:10.176 INFO:tasks.workunit.client.0.smithi129.stdout:test api_stat_pp on pid 92682 2023-12-07T23:47:10.183 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92682 2023-12-07T23:47:10.183 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.183 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.183 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_watch_notify 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.185 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_watch_notify' 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.186 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_misc: /' 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_misc.log 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_watch_notify 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.189 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_watch_notify 2023-12-07T23:47:10.190 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92718 2023-12-07T23:47:10.190 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_watch_notify on pid 92718' 2023-12-07T23:47:10.190 INFO:tasks.workunit.client.0.smithi129.stdout:test api_watch_notify on pid 92718 2023-12-07T23:47:10.203 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92718 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_watch_notify_pp 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_watch_notify_pp' 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_watch_notify_pp 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.209 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.211 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.212 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.215 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.216 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.218 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.218 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92552/exe 2023-12-07T23:47:10.218 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_watch_notify_pp 2023-12-07T23:47:10.218 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92750 2023-12-07T23:47:10.218 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_watch_notify_pp on pid 92750' 2023-12-07T23:47:10.218 INFO:tasks.workunit.client.0.smithi129.stdout:test api_watch_notify_pp on pid 92750 2023-12-07T23:47:10.234 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92750 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.235 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.236 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cmd 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cmd' 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92611/exe 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.238 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.239 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92532/exe 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.240 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.241 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.242 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.243 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.243 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.243 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.243 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.243 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.243 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.243 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.245 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.245 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_pool.log 2023-12-07T23:47:10.245 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.245 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-12-07T23:47:10.245 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.245 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.245 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.246 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.247 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-12-07T23:47:10.248 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.250 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.251 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.252 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.252 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.252 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.252 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.252 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-12-07T23:47:10.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] Global test environment set-up. 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] 12 tests from AsioRados 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.088+0000 7f654320c780 10 monclient: get_monmap_and_config 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.088+0000 7f654320c780 10 monclient: build_initial_monmap 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.088+0000 7f654320c780 10 monclient: monmap: 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 0 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 0.000000 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 0.000000 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:47:10.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.088+0000 7f654320c780 1 Processor -- start 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.088+0000 7f654320c780 1 -- start start 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.088+0000 7f654320c780 10 monclient: init 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient: _reopen_session rank -1 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 0x55c9f7b6d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): picked mon.noname-b con 0x7f65280030b0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7b6eb80 0x55c9f7ae0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): picked mon.noname-c con 0x55c9f7b6eb80 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c9f7adff80 0x55c9f7ae0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:10.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): picked mon.noname-a con 0x55c9f7adff80 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): start opening mon connection 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f7a58780 con 0x55c9f7adff80 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): start opening mon connection 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f79f2970 con 0x7f65280030b0 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): start opening mon connection 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c9f79f2610 con 0x55c9f7b6eb80 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): _renew_subs 2023-12-07T23:47:10.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.089+0000 7f654320c780 10 monclient(hunting): authenticate will time out at 1430.277777s 2023-12-07T23:47:10.256 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cmd 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cmd 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92801 2023-12-07T23:47:10.257 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cmd on pid 92801' 2023-12-07T23:47:10.258 INFO:tasks.workunit.client.0.smithi129.stdout: test api_cmd on pid 92801 2023-12-07T23:47:10.262 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92801 2023-12-07T23:47:10.262 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_pool: /' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cmd_pp 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cmd_pp' 2023-12-07T23:47:10.263 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.264 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cmd_pp 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_tier_pp: /' 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92595/exe 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cmd_pp 2023-12-07T23:47:10.267 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92817 2023-12-07T23:47:10.268 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cmd_pp on pid 92817 2023-12-07T23:47:10.270 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cmd_pp on pid 92817' 2023-12-07T23:47:10.270 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92817 2023-12-07T23:47:10.270 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.270 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.270 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.271 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.091+0000 7f653f89c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7b6eb80 0x55c9f7ae0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.091+0000 7f653f89c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7b6eb80 0x55c9f7ae0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:49950/0 (socket says 172.21.15.129:49950) 2023-12-07T23:47:10.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.091+0000 7f653f89c640 1 -- 172.21.15.129:0/3632560888 learned_addr learned my addr 172.21.15.129:0/3632560888 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.091+0000 7f653f89c640 10 monclient(hunting): get_auth_request con 0x55c9f7b6eb80 auth_method 0 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.091+0000 7f653f89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.091+0000 7f653f89c640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.091+0000 7f653f89c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.093+0000 7f653f89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.093+0000 7f653f89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.093+0000 7f653f89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.093+0000 7f654009d640 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 0x55c9f7b6d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 1 -- 172.21.15.129:0/3632560888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 msgr2=0x55c9f7b6d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 0x55c9f7b6d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 1 -- 172.21.15.129:0/3632560888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c9f7adff80 msgr2=0x55c9f7ae0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T23:47:10.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c9f7adff80 0x55c9f7ae0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 10 monclient: _finish_hunting 0 2023-12-07T23:47:10.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 1 monclient: found mon.noname-c 2023-12-07T23:47:10.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:47:10.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:47:10.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.095+0000 7f653f89c640 1 -- 172.21.15.129:0/3632560888 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9f79fe770 con 0x55c9f7b6eb80 2023-12-07T23:47:10.280 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92718/exe 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.281 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.282 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.284 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.285 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.286 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.106+0000 7f653f89c640 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7b6eb80 0x55c9f7ae0f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6530044460 tx=0x7f6530002fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9890f533c762e512 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 1 -- 172.21.15.129:0/3632560888 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530052640 con 0x55c9f7b6eb80 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: dump: 2023-12-07T23:47:10.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: _finish_auth 0 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: _check_auth_tickets 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:10.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 1 -- 172.21.15.129:0/3632560888 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65300527a0 con 0x55c9f7b6eb80 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 1 -- 172.21.15.129:0/3632560888 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530052b50 con 0x55c9f7b6eb80 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f653e89a640 10 monclient: dump: 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:47:10.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:47:10.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:47:10.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f654320c780 5 monclient: authenticate success, global_id 4340 2023-12-07T23:47:10.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f654320c780 10 monclient: get_monmap_and_config success 2023-12-07T23:47:10.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.113+0000 7f654320c780 4 set_mon_vals no callback set 2023-12-07T23:47:10.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 10 monclient: shutdown 2023-12-07T23:47:10.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:47:10.292 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2023-12-07T23:47:10.292 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_service 2023-12-07T23:47:10.293 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 1 -- 172.21.15.129:0/3632560888 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7b6eb80 msgr2=0x55c9f7ae0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7b6eb80 0x55c9f7ae0f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6530044460 tx=0x7f6530002fb0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 1 -- 172.21.15.129:0/3632560888 shutdown_connections 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c9f7adff80 0x55c9f7ae0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7b6eb80 0x55c9f7ae0f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 1 --2- 172.21.15.129:0/3632560888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 0x55c9f7b6d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.115+0000 7f654320c780 1 -- 172.21.15.129:0/3632560888 >> 172.21.15.129:0/3632560888 conn(0x55c9f7ae2180 msgr2=0x55c9f7b6df90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.116+0000 7f654320c780 1 -- 172.21.15.129:0/3632560888 shutdown_connections 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.116+0000 7f654320c780 1 -- 172.21.15.129:0/3632560888 wait complete. 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.116+0000 7f654320c780 10 monclient: build_initial_monmap 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.116+0000 7f654320c780 10 monclient: monmap: 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 0 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 0.000000 2023-12-07T23:47:10.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 0.000000 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.116+0000 7f654320c780 1 librados: starting msgr at 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.116+0000 7f654320c780 1 librados: starting objecter 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.116+0000 7f654320c780 1 Processor -- start 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 -- start start 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 librados: setting wanted keys 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 librados: calling monclient init 2023-12-07T23:47:10.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient: init 2023-12-07T23:47:10.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient: _reopen_session rank -1 2023-12-07T23:47:10.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T23:47:10.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7adff80 0x55c9f7b6c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:10.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): picked mon.noname-c con 0x55c9f7adff80 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_service' 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92682/exe 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_watch_notify: /' 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.309 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_snapshots: /' 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2023-12-07T23:47:10.311 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_service 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.312 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.313 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.314 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_stat_pp: /' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.315 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.316 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.317 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.318 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_service 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92909 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_service on pid 92909' 2023-12-07T23:47:10.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c9f7b6eb80 0x55c9f7b640c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto test api_service on pid 92909 2023-12-07T23:47:10.322 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92909 2023-12-07T23:47:10.322 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.322 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.322 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.322 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.322 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.322 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.323 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_service_pp 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_service_pp' 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stdout:rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:10.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): picked mon.noname-a con 0x55c9f7b6eb80 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 0x55c9f7aa19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): picked mon.noname-b con 0x7f65280030b0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): start opening mon connection 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f79d24e0 con 0x55c9f7b6eb80 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): start opening mon connection 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f7b680f0 con 0x7f65280030b0 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): start opening mon connection 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c9f7b64600 con 0x55c9f7adff80 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): _renew_subs 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.117+0000 7f654320c780 10 monclient(hunting): authenticate will time out at 1430.305927s 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.143+0000 7f654009d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7adff80 0x55c9f7b6c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:47:10.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.144+0000 7f654009d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7adff80 0x55c9f7b6c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:49986/0 (socket says 172.21.15.129:49986) 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.144+0000 7f654009d640 1 -- 172.21.15.129:0/1088732227 learned_addr learned my addr 172.21.15.129:0/1088732227 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.144+0000 7f654009d640 10 monclient(hunting): get_auth_request con 0x55c9f7adff80 auth_method 0 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.144+0000 7f654009d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.144+0000 7f654009d640 10 monclient(hunting): _init_auth method 2 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.144+0000 7f654009d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.145+0000 7f654009d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.145+0000 7f654009d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.145+0000 7f654009d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654089e640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 0x55c9f7aa19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:47:10.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2023-12-07T23:47:10.329 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-12-07T23:47:10.329 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.329 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_service_pp 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_service_pp 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92922 2023-12-07T23:47:10.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_test api_service_pp on pid 92922 2023-12-07T23:47:10.343 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_service_pp on pid 92922' 2023-12-07T23:47:10.343 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92922 2023-12-07T23:47:10.343 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.343 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.343 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.343 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_c_write_operations 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_c_write_operations' 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92648/exe 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.344 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.345 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.346 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92801/exe 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.347 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.348 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.349 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.350 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_stat.log 2023-12-07T23:47:10.351 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_c_write_operations 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.352 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.353 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.353 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_c_write_operations 2023-12-07T23:47:10.353 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.353 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92946 2023-12-07T23:47:10.353 INFO:tasks.workunit.client.0.smithi129.stdout:test api_c_write_operations on pid 92946 2023-12-07T23:47:10.368 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_c_write_operations on pid 92946' 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92946 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cmd.log 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cmd: /' 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.369 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_stat: /' 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92750/exe 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.373 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92817/exe 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.374 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.375 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_c_read_operations 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.376 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_c_read_operations' 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.377 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.378 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2023-12-07T23:47:10.379 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_c_read_operations 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.380 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.381 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.382 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_c_read_operations 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.383 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92998 2023-12-07T23:47:10.384 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_c_read_operations on pid 92998' 2023-12-07T23:47:10.384 INFO:tasks.workunit.client.0.smithi129.stdout:test api_c_read_operations on pid 92998 2023-12-07T23:47:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=92998 2023-12-07T23:47:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cls_remote_reads 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cls_remote_reads' 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cls_remote_reads 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.393 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92909/exe 2023-12-07T23:47:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cls_remote_reads 2023-12-07T23:47:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93021 2023-12-07T23:47:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cls_remote_reads on pid 93021' 2023-12-07T23:47:10.399 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cls_remote_reads on pid 93021 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93021 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.408 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.410 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s list_parallel 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' list_parallel' 2023-12-07T23:47:10.411 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92922/exe 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.412 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo list_parallel 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_service: /' 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.414 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.415 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_service.log 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=list_parallel 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93038 2023-12-07T23:47:10.416 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test list_parallel on pid 93038' 2023-12-07T23:47:10.417 INFO:tasks.workunit.client.0.smithi129.stdout:test list_parallel on pid 93038 2023-12-07T23:47:10.427 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93038 2023-12-07T23:47:10.427 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-12-07T23:47:10.427 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.427 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.427 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_service_pp: /' 2023-12-07T23:47:10.427 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s open_pools_parallel 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' open_pools_parallel' 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.428 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.429 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.430 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.431 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.432 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.433 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo open_pools_parallel 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92946/exe 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=open_pools_parallel 2023-12-07T23:47:10.434 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93082 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test open_pools_parallel on pid 93082' 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.435 INFO:tasks.workunit.client.0.smithi129.stdout:test open_pools_parallel on pid 93082 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93082 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.445 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.446 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s delete_pools_parallel 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-12-07T23:47:10.447 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2023-12-07T23:47:10.448 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' delete_pools_parallel' 2023-12-07T23:47:10.449 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.450 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo delete_pools_parallel 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=delete_pools_parallel 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93109 2023-12-07T23:47:10.452 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.453 INFO:tasks.workunit.client.0.smithi129.stdout:test delete_pools_parallel on pid 93109 2023-12-07T23:47:10.464 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test delete_pools_parallel on pid 93109' 2023-12-07T23:47:10.464 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93109 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.465 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.466 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s cls 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' cls' 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.467 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/92998/exe 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.468 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.469 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.470 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.471 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo cls 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.472 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.473 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=cls 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93143 2023-12-07T23:47:10.475 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test cls on pid 93143' 2023-12-07T23:47:10.476 INFO:tasks.workunit.client.0.smithi129.stdout:test cls on pid 93143 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93143 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.493 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93021/exe 2023-12-07T23:47:10.497 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.499 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.500 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.503 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s cmd 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.504 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' cmd' 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.505 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo cmd 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.506 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.507 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=cmd 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.508 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93186 2023-12-07T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test cmd on pid 93186' 2023-12-07T23:47:10.510 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.510 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.510 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.510 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.510 INFO:tasks.workunit.client.0.smithi129.stdout:test cmd on pid 93186 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93186 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93038/exe 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.519 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.520 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.521 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.522 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s handler_error 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' handler_error' 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo handler_error 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93143/exe 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.523 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.524 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_cls.log 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ list_parallel: /' 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.525 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.526 INFO:tasks.workunit.client.0.smithi129.stdout:test handler_error on pid 93215 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout:asio: 2023-12-07T23:47:10.149+0000 7f654009d640 1 -- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 msgr2=0x55c9f7aa19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280030b0 0x55c9f7aa19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 1 -- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c9f7b6eb80 msgr2=0x55c9f7b640c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c9f7b6eb80 0x55c9f7b640c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 10 monclient: _finish_hunting 0 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 1 monclient: found mon.noname-c 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.149+0000 7f654009d640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9f7a18760 con 0x55c9f7adff80 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.176+0000 7f654009d640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7adff80 0x55c9f7b6c800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f653400e670 tx=0x7f653400eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.176+0000 7f653d898640 10 client.?.objecter ms_handle_connect 0x55c9f7adff80 2023-12-07T23:47:10.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.176+0000 7f653d898640 10 client.?.objecter resend_mon_ops 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534009ab0 con 0x55c9f7adff80 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: dump: 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:47:10.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:47:10.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: _finish_auth 0 2023-12-07T23:47:10.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: _check_auth_tickets 2023-12-07T23:47:10.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=handler_error 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93215 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test handler_error on pid 93215' 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93215 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.538 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.539 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_list_parallel.log 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ cls: /' 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_cls 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.541 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s io 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' io' 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.542 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.543 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93082/exe 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.545 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.546 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo io 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93109/exe 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=io 2023-12-07T23:47:10.547 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.548 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.548 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93272 2023-12-07T23:47:10.548 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6534009c10 con 0x55ctest io on pid 93272 2023-12-07T23:47:10.554 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test io on pid 93272' 2023-12-07T23:47:10.554 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93272 2023-12-07T23:47:10.554 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.554 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.554 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.554 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s list 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' list' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.555 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo list 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-12-07T23:47:10.556 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.557 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.558 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.559 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.560 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=list 2023-12-07T23:47:10.560 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.560 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.560 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93297 2023-12-07T23:47:10.560 INFO:tasks.workunit.client.0.smithi129.stdout:test list on pid 93297 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test list on pid 93297' 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93297 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.573 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s misc 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.575 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' misc' 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.576 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.577 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.578 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.579 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo misc 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.580 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.581 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.582 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.583 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93186/exe 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.584 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.585 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_cmd 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout:9f7adff80 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: handle_config config(0 keys) v1 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653401b430 con 0x55c9f7adff80 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653d898640 10 monclient: dump: 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2023-12-07T23:44:30.440011+0000 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f653e89a640 4 set_mon_vals no callback set 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f654320c780 5 monclient: authenticate success, global_id 4355 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f654320c780 10 monclient: _renew_subs 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f654320c780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.301+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c9f79f1920 con 0x55c9f7adff80 2023-12-07T23:47:10.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.302+0000 7f654320c780 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.302+0000 7f654320c780 10 monclient: _renew_subs 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.302+0000 7f654320c780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.302+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9f79f1380 con 0x55c9f7adff80 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.302+0000 7f654320c780 1 librados: init done 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.302+0000 7f654320c780 10 librados: wait_for_osdmap waiting 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.364+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6534011400 con 0x55c9f7adff80 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.364+0000 7f653d898640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f652c06aac0 0x7f652c06cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.365+0000 7f653f89c640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f652c06aac0 0x7f652c06cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:47:10.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.365+0000 7f653f89c640 10 monclient: get_auth_request con 0x7f652c06aac0 auth_method 0 2023-12-07T23:47:10.593 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_cmd.log 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ cmd: /' 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.595 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.596 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.597 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=misc 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.598 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.599 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93405 2023-12-07T23:47:10.599 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.365+0000 7f653f89c640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f652c06aac0 0x7f652c06cf80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55c9f7b7db60 tx=0x7f653000c040 comp rtest misc on pid 93405 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test misc on pid 93405' 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93405 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s pool 2023-12-07T23:47:10.610 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' pool' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.611 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93272/exe 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.612 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.613 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93215/exe 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ io: /' 2023-12-07T23:47:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo pool 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_io 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_io.log 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.616 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_handler_error 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.617 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.618 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=pool 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93446 2023-12-07T23:47:10.619 INFO:tasks.workunit.client.0.smithi129.stdout:test pool on pid 93446 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test pool on pid 93446' 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93446 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_handler_error.log 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s read_operations 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' read_operations' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo read_operations 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.628 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ handler_error: /' 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.630 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=read_operations 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93465 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test read_operations on pid 93465' 2023-12-07T23:47:10.631 INFO:tasks.workunit.client.0.smithi129.stdout:test read_operations on pid 93465 2023-12-07T23:47:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93465 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s snapshots 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93297/exe 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' snapshots' 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.640 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.641 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_list.log 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_list 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.644 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.645 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo snapshots 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=snapshots 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93486 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test snapshots on pid 93486' 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ list: /' 2023-12-07T23:47:10.647 INFO:tasks.workunit.client.0.smithi129.stdout:test snapshots on pid 93486 2023-12-07T23:47:10.652 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93486 2023-12-07T23:47:10.652 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.652 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.652 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s watch_notify 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' watch_notify' 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93405/exe 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo watch_notify 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=watch_notify 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.656 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.657 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93495 2023-12-07T23:47:10.657 INFO:tasks.workunit.client.0.smithi129.stdout:test watch_notify on pid 93495 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test watch_notify on pid 93495' 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93495 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_misc 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_misc.log 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2023-12-07T23:47:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.671 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.672 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.673 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s write_operations 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ misc: /' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' write_operations' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo write_operations 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=write_operations 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93529 2023-12-07T23:47:10.678 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test write_operations on pid 93529' 2023-12-07T23:47:10.679 INFO:tasks.workunit.client.0.smithi129.stdout:test write_operations on pid 93529 2023-12-07T23:47:10.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=93529 2023-12-07T23:47:10.708 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ ret=0 2023-12-07T23:47:10.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T23:47:10.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92532 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92532 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.711 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2023-12-07T23:47:10.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93446/exe 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T23:47:10.715 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.716 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.717 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ pool: /' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_pool 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.719 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_pool.log 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T23:47:10.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93465/exe 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93486/exe 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_read_operations 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ read_operations: /' 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_snapshots 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_read_operations.log 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_snapshots.log 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ snapshots: /' 2023-12-07T23:47:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93495/exe 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_watch_notify 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_watch_notify.log 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2023-12-07T23:47:10.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T23:47:10.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ watch_notify: /' 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/93529/exe 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2023-12-07T23:47:10.736 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T23:47:10.737 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2023-12-07T23:47:10.737 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T23:47:10.737 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2023-12-07T23:47:10.737 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_write_operations 2023-12-07T23:47:10.737 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ write_operations: /' 2023-12-07T23:47:10.737 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_write_operations.log 2023-12-07T23:47:11.799 INFO:tasks.workunit.client.0.smithi129.stdout:x=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:47:11.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.365+0000 7f653d898640 10 client.4355.objecter ms_handle_connect 0x7f652c06aac0 2023-12-07T23:47:11.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f653405a3b0 con 0x55c9f7adff80 2023-12-07T23:47:11.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(21..21 src has 1..21) v4 2023-12-07T23:47:11.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T23:47:11.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding full epoch 21 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f654320c780 10 librados: wait_for_osdmap done waiting 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f654320c780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.384+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55c9f7a51760 con 0x55c9f7adff80 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.517+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6534055ff0 con 0x55c9f7adff80 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.517+0000 7f653d898640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.517+0000 7f653e099640 10 client.4355.objecter _get_latest_version latest 21, have it 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.517+0000 7f654320c780 10 client.4355.objecter create_pool name=ceph_test_rados_api_asio 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.517+0000 7f654320c780 10 client.4355.objecter pool_op_submit 1 2023-12-07T23:47:11.800 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.517+0000 7f654320c780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:10.517+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55c9f7a62810 con 0x55c9f7adff80 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v22) v1 ==== 43+0+0 (secure 0 0 0) 0x7f6534020030 con 0x55c9f7adff80 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 pool_op_reply(tid 1 (0) Success v22) v1 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 10 client.4355.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v22) v1 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 10 client.4355.objecter have request 1 at 0x55c9f7aa1770 Op: create 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 20 client.4355.objecter waiting for client to reach epoch 22 before calling back 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:11.801 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=22}) v3 -- 0x7f652c070800 con 0x55c9f7adff80 2023-12-07T23:47:11.873 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.795+0000 7f653d898640 10 client.4355.objecter done 2023-12-07T23:47:11.873 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.860+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 5092+0+0 (secure 0 0 0) 0x7f6534059cb0 con 0x55c9f7adff80 2023-12-07T23:47:11.873 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.860+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(22..22 src has 1..22) v4 2023-12-07T23:47:11.873 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.860+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [22,22] > 21 2023-12-07T23:47:11.873 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.860+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 22 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.860+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.860+0000 7f654320c780 10 librados: set snap read head -> 1 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.860+0000 7f654320c780 10 librados: writefull oid=exist nspace= 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7b61eb0 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 22 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x55c9f7b62860 0x55c9f7c205a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 2 osd.1 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:11.874 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 2 2023-12-07T23:47:11.875 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 15 client.4355.objecter _send_op 2 to 15.1 on osd.1 2023-12-07T23:47:11.875 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:2 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) v8 -- 0x55c9f7aa1ef0 con 0x55c9f7b62860 2023-12-07T23:47:11.875 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:11.875 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.861+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:11.875 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.867+0000 7f654089e640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x55c9f7b62860 0x55c9f7c205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T23:47:11.875 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.867+0000 7f654089e640 10 monclient: get_auth_request con 0x55c9f7b62860 auth_method 0 2023-12-07T23:47:11.875 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.870+0000 7f654089e640 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x55c9f7b62860 0x55c9f7c205a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T23:47:14.190 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:10.662 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.254 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:10.699 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.256 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:10.701 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.266 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:10.739 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.305 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:10.777 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.328 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:10.773 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.342 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:10.814 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.398 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:10.869 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:14.480 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:10.953 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:20.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:11.870+0000 7f653d898640 10 client.4355.objecter ms_handle_connect 0x55c9f7b62860 2023-12-07T23:47:20.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:13.117+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:13.117+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:20.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:13.117+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:20.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:13.118+0000 7f653c896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:47:20.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:14.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:14.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:20.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:14.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:20.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:15.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:15.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:20.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:15.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:20.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:15.303+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:20.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:16.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:16.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:20.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:16.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:20.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:20.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:20.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.180+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 1 ==== osd_map(23..23 src has 1..23) v4 ==== 12808+0+0 (crc 0 0 0) 0x7f65380026c0 con 0x55c9f7b62860 2023-12-07T23:47:20.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.180+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(23..23 src has 1..23) v4 2023-12-07T23:47:20.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.180+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [23,23] > 22 2023-12-07T23:47:20.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.180+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 23 2023-12-07T23:47:20.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.181+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:20.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.181+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 23 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.181+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.181+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:17.181+0000 7f653d898640 20 client.4355.objecter 2 15.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:18.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:18.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:18.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:19.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:19.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:19.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:20.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:20.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:20.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:22.661 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] Global test environment set-up. 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] 4 tests from LibRadosService 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5901 ms) 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.RegisterLate (758 ms) 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2023-12-07T23:47:22.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 20 pool(s) do not have an application enabled 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: x(active, since 116s), standbys: y 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 8 osds: 8 up (since 72s), 8 in (since 116s) 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 36 pools, 1097 pgs 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 81.222% pgs unknown 2023-12-07T23:47:22.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 17.958% pgs not active 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 891 unknown 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 147 creating+peering 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 50 creating+activating 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 9 active+clean 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 16 pool(s) do not have an application enabled 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2023-12-07T23:47:22.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: x(active, since 118s), standbys: y 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 118s) 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: foo: 9 portals active (1 hosts, 3 zones) 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 35 pools, 1065 pgs 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 66.009% pgs unknown 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 33.146% pgs not active 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 703 unknown 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 309 creating+peering 2023-12-07T23:47:22.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 44 creating+activating 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 9 active+clean 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 19 pool(s) do not have an application enabled 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 8 osds: 8 up (since 77s), 8 in (since 2m) 2023-12-07T23:47:22.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-12-07T23:47:22.667 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-12-07T23:47:22.667 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:22.667 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2023-12-07T23:47:22.667 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 39 pools, 1169 pgs 2023-12-07T23:47:25.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 3 obj api_asio: 2023-12-07T23:47:20.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:20.304+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:21.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:21.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:21.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:22.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:22.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:22.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:23.118+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:23.118+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:23.118+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:23.119+0000 7f653c896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:24.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:25.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:24.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:24.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.305+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.305+0000 7f6542301640 2 client.4355.objecter tid 2 on osd.1 is laggy 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.305+0000 7f6542301640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.305+0000 7f6542301640 10 monclient: _renew_subs 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.305+0000 7f6542301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.305+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=24}) v3 -- 0x7f6518002f90 con 0x55c9f7adff80 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.305+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- ping magic: 0 v1 -- 0x7f6518003320 con 0x55c9f7b62860 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 9 ==== osd_map(24..30 src has 1..30) v4 ==== 12294+0+0 (secure 0 0 0) 0x7f653405de00 con 0x55c9f7adff80 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(24..30 src has 1..30) v4 2023-12-07T23:47:25.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [24,30] > 23 2023-12-07T23:47:25.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 24 2023-12-07T23:47:25.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:25.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 24 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:25.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:27.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 25 2023-12-07T23:47:27.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 25 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 26 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 26 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 27 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 27 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 28 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 28 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:27.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 29 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 29 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 30 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 30 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:25.310+0000 7f653d898640 20 client.4355.objecter 2 15.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:26.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:26.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:26.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:27.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:27.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:27.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:27.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:28.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:28.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:28.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:29.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:29.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:29.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:30.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.306+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.306+0000 7f6542301640 2 client.4355.objecter tid 2 on osd.1 is laggy 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.306+0000 7f6542301640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.306+0000 7f6542301640 10 monclient: _renew_subs 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.306+0000 7f6542301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.306+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=31}) v3 -- 0x7f65180034e0 con 0x55c9f7adff80 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.306+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- ping magic: 0 v1 -- 0x7f6518003720 con 0x55c9f7b62860 2023-12-07T23:47:30.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 10 ==== osd_map(31..34 src has 1..34) v4 ==== 3614+0+0 (secure 0 0 0) 0x7f6534061fb0 con 0x55c9f7adff80 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(31..34 src has 1..34) v4 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [31,34] > 30 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 31 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 31 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 32 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 32 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 33 2023-12-07T23:47:30.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:33.932 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.obj pool: Running main() from gmock_main.cc 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [==========] Running 6 tests from 1 test suite. 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] Global test environment set-up. 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] 6 tests from NeoRadosPools 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolList (4705 ms) 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3378 ms) 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4060 ms) 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5988 ms) 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (3229 ms) 2023-12-07T23:47:33.933 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2023-12-07T23:47:33.934 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2107 ms) 2023-12-07T23:47:33.934 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] 6 tests from NeoRadosPools (23467 ms total) 2023-12-07T23:47:33.934 INFO:tasks.workunit.client.0.smithi129.stdout: pool: 2023-12-07T23:47:33.934 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] Global test environment tear-down 2023-12-07T23:47:33.934 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [==========] 6 tests from 1 test suite ran. (23467 ms total) 2023-12-07T23:47:33.934 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ PASSED ] 6 tests. 2023-12-07T23:47:34.171 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:30.644 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.179 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:30.652 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.251 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:30.723 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.257 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:30.729 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.268 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:30.713 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.275 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:30.721 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.421 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:30.894 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.464 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:30.908 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:34.478 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:30.951 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:35.528 INFO:tasks.workunit.client.0.smithi129.stdout:ecter _calc_target epoch 33 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 34 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 34 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:30.309+0000 7f653d898640 20 client.4355.objecter 2 15.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:31.119+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:31.119+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:35.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:31.119+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:35.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:32.120+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:35.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:32.120+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:35.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:32.120+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:35.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:33.120+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:35.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:33.120+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:35.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:33.120+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:35.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:33.120+0000 7f653c896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:47:35.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:34.120+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:35.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:34.120+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:35.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:34.120+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:35.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.120+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:35.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.120+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.120+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.307+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.307+0000 7f6542301640 2 client.4355.objecter tid 2 on osd.1 is laggy 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.307+0000 7f6542301640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.307+0000 7f6542301640 10 monclient: _renew_subs 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.307+0000 7f6542301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.307+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=35}) v3 -- 0x7f65180038b0 con 0x55c9f7adff80 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.307+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- ping magic: 0 v1 -- 0x7f6518003b00 con 0x55c9f7b62860 2023-12-07T23:47:35.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 11 ==== osd_map(35..38 src has 1..38) v4 ==== 2904+0+0 (secure 0 0 0) 0x7f6534059a80 con 0x55c9f7adff80 2023-12-07T23:47:37.192 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-1 handler_error: Running main() from gmock_main.cc 2023-12-07T23:47:37.193 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2023-12-07T23:47:37.193 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] Global test environment set-up. 2023-12-07T23:47:37.193 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] 1 test from neocls_handler_error 2023-12-07T23:47:37.193 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2023-12-07T23:47:37.193 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (26775 ms) 2023-12-07T23:47:37.194 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] 1 test from neocls_handler_error (26775 ms total) 2023-12-07T23:47:37.194 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: 2023-12-07T23:47:37.194 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] Global test environment tear-down 2023-12-07T23:47:37.194 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [==========] 1 test from 1 test suite ran. (26775 ms total) 2023-12-07T23:47:37.194 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [ PASSED ] 1 test. 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout:2-07T23:47:35.525+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(35..38 src has 1..38) v4 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [35,38] > 34 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 35 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 35 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 36 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 36 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 37 2023-12-07T23:47:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 37 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 38 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 38 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:35.525+0000 7f653d898640 20 client.4355.objecter 2 15.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:36.120+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:36.120+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:36.120+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:37.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:37.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:37.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:38.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:38.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:38.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:39.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:39.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:39.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:41.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:41.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:41.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.308+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:41.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.308+0000 7f6542301640 2 client.4355.objecter tid 2 on osd.1 is laggy 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.308+0000 7f6542301640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.308+0000 7f6542301640 10 monclient: _renew_subs 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.308+0000 7f6542301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.308+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=39}) v3 -- 0x7f6518003c90 con 0x55c9f7adff80 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.308+0000 7f6542301640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- ping magic: 0 v1 -- 0x7f6518003e20 con 0x55c9f7b62860 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.311+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 12 ==== osd_map(39..40 src has 1..40) v4 ==== 1327+0+0 (secure 0 0 0) 0x7f6534057df0 con 0x55c9f7adff80 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.311+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(39..40 src has 1..40) v4 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.311+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [39,40] > 38 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.311+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 39 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 39 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:41.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 40 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 10 client.4355.objecter checking op 2 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 40 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:40.312+0000 7f653d898640 20 client.4355.objecter 2 15.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 2 ==== osd_op_reply(2 exist [writefull 0~5] v23'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(2 exist [writefull 0~5] v23'1 uv1 ondisk = 0) v8 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 2 ondisk uv 1 in 15.1 attempt 0 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 2 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 15 client.4355.objecter _finish_op 2 2023-12-07T23:47:41.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 2 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 10 librados: Objecter returned from writefull r=0 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7b61eb0 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 40 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [read 0~256] tid 3 osd.1 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 3 2023-12-07T23:47:41.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 15 client.4355.objecter _send_op 3 to 15.1 on osd.1 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:3 15.1 15:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v8 -- 0x55c9f7a9f8e0 con 0x55c9f7b62860 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7a9fc60 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 40 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 10 client.4355.objecter _op_submit oid noexist '@15' '@15' [read 0~256] tid 4 osd.1 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 4 2023-12-07T23:47:41.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 15 client.4355.objecter _send_op 4 to 15.1 on osd.1 2023-12-07T23:47:41.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:4 15.1 15:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v8 -- 0x55c9f7aa0100 con 0x55c9f7b api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-12-07T23:47:41.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] Global test environment set-up. 2023-12-07T23:47:41.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-12-07T23:47:41.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5319 ms) 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (80 ms) 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20058 ms) 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: attempt 0 of 20 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: failed to deregister: 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: { 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "epoch": 10, 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "modified": "2023-12-07T23:47:37.574793+0000", 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "services": { 2023-12-07T23:47:41.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "laundry": { 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "daemons": { 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "summary": "", 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "close-test-pid93032": { 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_epoch": 10, 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_stamp": "2023-12-07T23:47:35.801229+0000", 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "gid": 4613, 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "addr": "172.21.15.129:0/2217591690", 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "metadata": { 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "arch": "x86_64", 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_release": "reef", 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-83-gc56ad3ea (c56ad3eaf0784383bf22c04e35d26e15c4baeafe) reef (dev)", 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version_short": "19.0.0-83-gc56ad3ea", 2023-12-07T23:47:41.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro": "centos", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_version": "9", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "foo": "bar", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "hostname": "smithi129", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_version": "5.14.0-390.el9.x86_64", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_total_kb": "32488608", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "os": "Linux", 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "this": "that" 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: }, 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "task_status": {} 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: }, 2023-12-07T23:47:41.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "pid93018": { 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_epoch": 9, 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_stamp": "2023-12-07T23:47:22.977769+0000", 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "gid": 4729, 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "addr": "172.21.15.129:0/1922275202", 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "metadata": { 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "arch": "x86_64", 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_release": "reef", 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-83-gc56ad3ea (c56ad3eaf0784383bf22c04e35d26e15c4baeafe) reef (dev)", 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version_short": "19.0.0-83-gc56ad3ea", 2023-12-07T23:47:41.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-07T23:47:42.962 INFO:tasks.workunit.client.0.smithi129.stdout: api_seects, 449 KiB 2023-12-07T23:47:42.962 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-07T23:47:42.962 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 44.311% pgs unknown 2023-12-07T23:47:42.962 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 53.037% pgs not active 2023-12-07T23:47:42.962 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 518 unknown 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 479 creating+peering 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 141 creating+activating 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 31 active+clean 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: io: 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: client: 212 B/s wr, 0 op/s rd, 0 op/s wr 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5765 ms) 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.Status 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.Status (20243 ms) 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] 4 tests from LibRadosService (32667 ms total) 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] Global test environment tear-down 2023-12-07T23:47:42.963 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32670 ms total) 2023-12-07T23:47:42.964 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ PASSED ] 4 tests. 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout:62860 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.049+0000 7f654320c780 5 client.4355.objecter 2 in flight 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:41.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:42.120+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:42.120+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:42.120+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.120+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.120+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:43.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.120+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.120+0000 7f653c896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 3 ==== osd_map(41..41 src has 1..42) v4 ==== 1051+0+0 (crc 0 0 0) 0x7f6538013410 con 0x55c9f7b62860 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(41..41 src has 1..42) v4 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [41,41] > 40 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 41 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 10 client.4355.objecter checking op 3 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 41 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 10 client.4355.objecter checking op 4 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 41 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:43.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 20 client.4355.objecter 3 15.d3a59201 osd.1 exist [read 0~256] 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f653d898640 20 client.4355.objecter 4 15.a7bbaa1 osd.1 noexist [read 0~256] 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 3 ondisk uv 1 in 15.1 attempt 0 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 5 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 3 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 15 client.4355.objecter _finish_op 3 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 3 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 5 client.4355.objecter 1 in flight 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T23:47:43.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 4 ondisk uv 0 in 15.1 attempt 0 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 4 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 15 client.4355.objecter _finish_op 4 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 4 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2552 ms) 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7a9fc60 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 41 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:43.607 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:43.608 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:43.608 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [read 0~256] tid 5 osd.1 2023-12-07T23:47:43.608 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:43.608 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 5 2023-12-07T23:47:43.608 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 15 client.4355.objecter _send_op 5 to 15.1 on osd.1 2023-12-07T23:47:43.608 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:5 15.1 15:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55c9f7aa0460 con 0x55c9f7b62860 2023-12-07T23:47:43.608 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:44.205 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:47:44.189+0000 ea50640 -1 osd.6 41 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:2 14.16 14:6c367cf0:test-rados-api-smithi129-92858-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:45.319 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:47:45.235+0000 ea50640 -1 osd.6 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:2 14.16 14:6c367cf0:test-rados-api-smithi129-92858-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7b61eb0 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 41 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 10 client.4355.objecter _op_submit oid noexist '@15' '@15' [read 0~256] tid 6 osd.1 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 6 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 15 client.4355.objecter _send_op 6 to 15.1 on osd.1 2023-12-07T23:47:45.481 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:6 15.1 15:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55c9f7c20ae0 con 0x55c9f7b62860 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:43.600+0000 7f654320c780 5 client.4355.objecter 2 in flight 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:44.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:44.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:44.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.308+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 5 ondisk uv 1 in 15.1 attempt 0 2023-12-07T23:47:45.482 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 5 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 5 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 15 client.4355.objecter _finish_op 5 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 5 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 5 client.4355.objecter 1 in flight 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 6 ==== osd_map(42..42 src has 1..42) v4 ==== 684+0+0 (crc 0 0 0) 0x7f6538002e10 con 0x55c9f7b62860 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(42..42 src has 1..42) v4 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 6 ondisk uv 0 in 15.1 attempt 0 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:47:45.483 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 6 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 15 client.4355.objecter _finish_op 6 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 6 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [42,42] > 41 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 42 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1877 ms) 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7b61eb0 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 42 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:45.484 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:45.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:45.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:45.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [read 0~256] tid 7 osd.1 2023-12-07T23:47:45.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:45.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 7 2023-12-07T23:47:45.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 15 client.4355.objecter _send_op 7 to 15.1 on osd.1 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:7 15.1 15:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55c9f7a9fc60 con 0x55c9f7b62860 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c614d0 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 42 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 10 client.4355.objecter _op_submit oid noexist '@15' '@15' [read 0~256] tid 8 osd.1 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:45.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 8 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 15 client.4355.objecter _send_op 8 to 15.1 on osd.1 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:8 15.1 15:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55c9f7c61970 con 0x55c9f7b62860 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.477+0000 7f654320c780 5 client.4355.objecter 2 in flight 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 9 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 7 ondisk uv 1 in 15.1 attempt 0 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 5 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 7 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 15 client.4355.objecter _finish_op 7 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 7 2023-12-07T23:47:45.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 5 client.4355.objecter 1 in flight 2023-12-07T23:47:45.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 10 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:45.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T23:47:46.305 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:47:46.298+0000 ea50640 -1 osd.6 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:2 14.16 14:6c367cf0:test-rados-api-smithi129-92858-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:47.393 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:47:47.357+0000 ea50640 -1 osd.6 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:2 14.16 14:6c367cf0:test-rados-api-smithi129-92858-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 8 ondisk uv 0 in 15.1 attempt 0 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.792+0000 7f654089e640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 8 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654089e640 15 client.4355.objecter _finish_op 8 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 8 2023-12-07T23:47:48.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (316 ms) 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 librados: queue_aio_write 0x55c9f7a51480 completion 0x55c9f79f0d40 write_seq 1 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7b61eb0 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 42 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [write 0~5 in=5b] tid 9 osd.1 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 9 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 15 client.4355.objecter _send_op 9 to 15.1 on osd.1 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:9 15.1 15:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55c9f7c21e50 con 0x55c9f7b62860 2023-12-07T23:47:48.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:45.793+0000 7f654320c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:46.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:46.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:46.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:47.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:47.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:48.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:47.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:48.908 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:47:48.890+0000 ea50640 -1 osd.6 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:2 14.16 14:6c367cf0:test-rados-api-smithi129-92858-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:49.512 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:47:49.492+0000 ea50640 -1 osd.3 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4346.0:2 8.15 8:adff54c6:test-rados-api-smithi129-92574-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:50.285 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:47:50.159+0000 ea50640 -1 osd.6 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:2 14.16 14:6c367cf0:test-rados-api-smithi129-92858-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:51.889 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:47:51.769+0000 ea50640 -1 osd.6 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4358.0:2 14.16 14:6c367cf0:test-rados-api-smithi129-92858-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-12-07T23:47:52.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_arvice_pp: "distro": "centos", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_version": "9", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "foo": "bar", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "hostname": "smithi129", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_version": "5.14.0-390.el9.x86_64", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_total_kb": "32488608", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "os": "Linux", 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "this": "that" 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: }, 2023-12-07T23:47:52.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "task_status": {} 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: attempt 1 of 20 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: failed to deregister: 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: { 2023-12-07T23:47:52.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "epoch": 13, 2023-12-07T23:47:52.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "modified": "2023-12-07T23:47:43.578356+0000", 2023-12-07T23:47:52.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "services": { 2023-12-07T23:47:52.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "laundry": { 2023-12-07T23:47:52.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "daemons": { 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "summary": "", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "close-test-pid93032": { 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_epoch": 12, 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_stamp": "2023-12-07T23:47:41.572806+0000", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "gid": 4581, 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "addr": "172.21.15.129:0/3509555369", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "metadata": { 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "arch": "x86_64", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_release": "reef", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-83-gc56ad3ea (c56ad3eaf0784383bf22c04e35d26e15c4baeafe) reef (dev)", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version_short": "19.0.0-83-gc56ad3ea", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro": "centos", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-07T23:47:52.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_version": "9", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "foo": "bar", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "hostname": "smithi129", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_version": "5.14.0-390.el9.x86_64", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_total_kb": "32488608", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "os": "Linux", 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "this": "that" 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: }, 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "task_status": {} 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: attempt 2 of 20 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: failed to deregister: 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: { 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "epoch": 15, 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "modified": "2023-12-07T23:47:47.579825+0000", 2023-12-07T23:47:52.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "services": { 2023-12-07T23:47:52.534 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:47:52.486+0000 ea50640 -1 osd.2 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:47:53.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_servisio: 2023-12-07T23:47:48.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:53.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:48.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:53.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:48.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:53.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:49.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:53.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:49.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:49.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:50.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:50.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:50.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:50.309+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:51.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:51.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:51.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:52.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:52.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:52.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:53.364 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.121+0000 7f653c896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 11 ==== osd_op_reply(9 exist [write 0~5] v42'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(9 exist [write 0~5] v42'2 uv2 ondisk = 0) v8 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 9 ondisk uv 2 in 15.1 attempt 0 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 9 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 15 client.4355.objecter _finish_op 9 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 9 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654089e640 20 librados: complete_aio_write 0x55c9f79f0d40 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (7567 ms) 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-07T23:47:53.365 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 librados: queue_aio_write 0x55c9f7a51480 completion 0x55c9f79e0660 write_seq 2 2023-12-07T23:47:53.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7b61eb0 2023-12-07T23:47:53.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 42 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [write 0~5 in=5b] tid 10 osd.1 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 10 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 15 client.4355.objecter _send_op 10 to 15.1 on osd.1 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:10 15.1 15:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v8 -- 0x55c9f7c21e50 con 0x55c9f7b62860 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.360+0000 7f654320c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 12 ==== osd_map(43..48 src has 1..48) v4 ==== 7854+0+0 (crc 0 0 0) 0x7f6538002e10 con 0x55c9f7b62860 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(43..48 src has 1..48) v4 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [43,48] > 42 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 43 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 43 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.447+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 44 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 44 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 45 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 45 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 46 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 46 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 47 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 47 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 48 2023-12-07T23:47:53.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 48 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=49}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.448+0000 7f653d898640 20 client.4355.objecter 10 15.d3a59201 osd.1 exist [write 0~5 in=5b] 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 13 ==== osd_map(49..50 src has 1..50) v4 ==== 2325+0+0 (secure 0 0 0) 0x7f65340563c0 con 0x55c9f7adff80 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(49..50 src has 1..50) v4 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [49,50] > 48 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 49 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 49 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:53.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 50 2023-12-07T23:47:53.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:53.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 50 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:53.540 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:47:53.529+0000 ea50640 -1 osd.2 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:47:54.772 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:47:54.650+0000 ea50640 -1 osd.2 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=51}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:53.472+0000 7f653d898640 20 client.4355.objecter 10 15.d3a59201 osd.1 exist [write 0~5 in=5b] 2023-12-07T23:47:55.083 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 14 ==== osd_map(51..51 src has 1..51) v4 ==== 1387+0+0 (secure 0 0 0) 0x7f65340563c0 con 0x55c9f7adff80 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(51..51 src has 1..51) v4 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [51,51] > 50 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 51 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 51 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=52}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.081+0000 7f653d898640 20 client.4355.objecter 10 15.d3a59201 osd.1 exist [write 0~5 in=5b] 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:55.084 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:54.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:55.085 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.078+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 15 ==== osd_map(52..52 src has 1..52) v4 ==== 1383+0+0 (secure 0 0 0) 0x7f6534083080 con 0x55c9f7adff80 2023-12-07T23:47:55.085 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.078+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(52..52 src has 1..52) v4 2023-12-07T23:47:55.085 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.078+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [52,52] > 51 2023-12-07T23:47:55.085 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.078+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 52 2023-12-07T23:47:55.085 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 10 client.4355.objecter checking op 10 2023-12-07T23:47:55.413 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:51.857 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.414 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:51.886 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.418 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:51.891 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.424 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:51.869 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.498 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:51.970 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.501 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:51.974 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.582 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:02:52.027 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.626 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:02:52.099 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.665 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:02:52.138 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:47:55.863 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:47:55.824+0000 ea50640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:47:56.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 52 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:56.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:56.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:56.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:56.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:56.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=53}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:47:56.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.079+0000 7f653d898640 20 client.4355.objecter 10 15.d3a59201 osd.1 exist [write 0~5 in=5b] 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.121+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.121+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.121+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:55.310+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 13 ==== osd_op_reply(10 exist [write 0~5] v48'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(10 exist [write 0~5] v48'3 uv3 ondisk = 0) v8 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 10 ondisk uv 3 in 15.1 attempt 0 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 10 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 15 client.4355.objecter _finish_op 10 2023-12-07T23:47:56.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 10 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.511+0000 7f654089e640 20 librados: complete_aio_write 0x55c9f79e0660 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3152 ms) 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 20 librados: queue_aio_write 0x55c9f7a51480 completion 0x55c9f7b61eb0 write_seq 3 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c445f0 2023-12-07T23:47:56.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 52 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:57.070 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:47:56.946+0000 ea50640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:47:57.818 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T2ce_pp: "laundry": { 2023-12-07T23:47:57.818 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "daemons": { 2023-12-07T23:47:57.818 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "summary": "", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "close-test-pid93032": { 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_epoch": 15, 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "start_stamp": "2023-12-07T23:47:47.192151+0000", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "gid": 4789, 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "addr": "172.21.15.129:0/1173269101", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "metadata": { 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "arch": "x86_64", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_release": "reef", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-83-gc56ad3ea (c56ad3eaf0784383bf22c04e35d26e15c4baeafe) reef (dev)", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "ceph_version_short": "19.0.0-83-gc56ad3ea", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro": "centos", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "distro_version": "9", 2023-12-07T23:47:57.819 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "foo": "bar", 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "hostname": "smithi129", 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023", 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "kernel_version": "5.14.0-390.el9.x86_64", 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "mem_total_kb": "32488608", 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "os": "Linux", 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "this": "that" 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: }, 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: "task_status": {} 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: } 2023-12-07T23:47:57.820 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: 2023-12-07T23:47:57.821 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: attempt 3 of 20 2023-12-07T23:47:57.821 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (22093 ms) 2023-12-07T23:47:57.821 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (47551 ms total) 2023-12-07T23:47:57.821 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: 2023-12-07T23:47:57.821 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] Global test environment tear-down 2023-12-07T23:47:57.821 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (47551 ms total) 2023-12-07T23:47:57.821 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout:3:47:56.512+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [write 0~5 in=5b] tid 11 osd.1 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 11 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 15 client.4355.objecter _send_op 11 to 15.1 on osd.1 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:11 15.1 15:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) v8 -- 0x55c9f7b61fc0 con 0x55c9f7b62860 2023-12-07T23:47:58.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.512+0000 7f654320c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 16 ==== osd_map(53..53 src has 1..53) v4 ==== 628+0+0 (secure 0 0 0) 0x7f6534083070 con 0x55c9f7adff80 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(53..53 src has 1..53) v4 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [53,53] > 52 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 53 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 10 client.4355.objecter checking op 11 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 53 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=54}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:58.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:56.651+0000 7f653d898640 20 client.4355.objecter 11 15.d3a59201 osd.1 exist [write 0~5 in=5b] 2023-12-07T23:47:58.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:57.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:58.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:57.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:58.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:57.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:58.109 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:47:58.043+0000 ea50640 -1 osd.2 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:47:59.103 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 17 ==== osd_map(54..54 src has 1..54) v4 ==== 675+0+0 (secure 0 0 0) 0x7f6534059f10 con 0x55c9f7adff80 2023-12-07T23:47:59.103 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(54..54 src has 1..54) v4 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [54,54] > 53 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 54 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 10 client.4355.objecter checking op 11 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 54 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.039+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=55}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.040+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.040+0000 7f653d898640 20 client.4355.objecter 11 15.d3a59201 osd.1 exist [write 0~5 in=5b] 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:47:59.104 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:58.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 18 ==== osd_map(55..55 src has 1..55) v4 ==== 312+0+0 (secure 0 0 0) 0x7f6534059f10 con 0x55c9f7adff80 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(55..55 src has 1..55) v4 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [55,55] > 54 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 55 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 10 client.4355.objecter checking op 11 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 55 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=56}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.047+0000 7f653d898640 20 client.4355.objecter 11 15.d3a59201 osd.1 exist [write 0~5 in=5b] 2023-12-07T23:47:59.105 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 14 ==== osd_op_reply(11 exist [write 0~5] v52'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(11 exist [write 0~5] v52'4 uv4 ondisk = 0) v8 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 11 ondisk uv 4 in 15.1 attempt 0 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 11 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 15 client.4355.objecter _finish_op 11 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 11 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654089e640 20 librados: complete_aio_write 0x55c9f7b61eb0 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2588 ms) 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c20e40 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 55 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:59.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [read 0~0] tid 12 osd.1 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 12 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 15 client.4355.objecter _send_op 12 to 15.1 on osd.1 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:12 15.1 15:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x55c9f7b61fc0 con 0x55c9f7b62860 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c212e0 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 55 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:47:59.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:47:59.624 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:47:59.595+0000 ea50640 -1 osd.4 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:47:59.740 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:47:59.718+0000 ea50640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:00.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:00.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 10 client.4355.objecter _op_submit oid noexist '@15' '@15' [read 0~0] tid 13 osd.1 2023-12-07T23:48:00.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:00.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 13 2023-12-07T23:48:00.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 15 client.4355.objecter _send_op 13 to 15.1 on osd.1 2023-12-07T23:48:00.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:13 15.1 15:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x55c9f7c21780 con 0x55c9f7b62860 2023-12-07T23:48:00.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.100+0000 7f654320c780 5 client.4355.objecter 2 in flight 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:47:59.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 19 ==== osd_map(56..56 src has 1..56) v4 ==== 680+0+0 (secure 0 0 0) 0x7f6534061b20 con 0x55c9f7adff80 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(56..56 src has 1..56) v4 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [56,56] > 55 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 56 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 10 client.4355.objecter checking op 12 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 56 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:00.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 10 client.4355.objecter checking op 13 2023-12-07T23:48:00.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 56 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:00.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:48:00.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:48:00.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:48:00.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:48:00.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=57}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:48:00.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:48:01.039 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:00.917+0000 ea50640 -1 osd.2 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:01.108 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:01.026+0000 ea50640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:01.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 20 client.4355.objecter 12 15.d3a59201 osd.1 exist [read 0~0] 2023-12-07T23:48:01.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.184+0000 7f653d898640 20 client.4355.objecter 13 15.a7bbaa1 osd.1 noexist [read 0~0] 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:00.311+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 20 ==== osd_map(57..57 src has 1..57) v4 ==== 668+0+0 (secure 0 0 0) 0x7f6534049060 con 0x55c9f7adff80 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(57..57 src has 1..57) v4 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [57,57] > 56 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 57 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 10 client.4355.objecter checking op 12 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 57 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:01.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 10 client.4355.objecter checking op 13 2023-12-07T23:48:01.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 57 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:01.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:48:01.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:48:01.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:48:01.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:48:01.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.220+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=58}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.221+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.221+0000 7f653d898640 20 client.4355.objecter 12 15.d3a59201 osd.1 exist [read 0~0] 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.221+0000 7f653d898640 20 client.4355.objecter 13 15.a7bbaa1 osd.1 noexist [read 0~0] 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 15 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 12 ondisk uv 4 in 15.1 attempt 0 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (2145 ms) 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 5 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 12 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 15 client.4355.objecter _finish_op 12 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:48:01.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 12 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 5 client.4355.objecter 1 in flight 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 16 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 13 ondisk uv 0 in 15.1 attempt 0 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 13 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 15 client.4355.objecter _finish_op 13 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 13 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c20e40 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 57 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:01.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [read 0~0] tid 14 osd.1 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 14 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 15 client.4355.objecter _send_op 14 to 15.1 on osd.1 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:14 15.1 15:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x55c9f7c212e0 con 0x55c9f7b62860 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:01.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:48:02.133 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:02.027+0000 ea50640 -1 osd.2 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:02.367 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c21890 2023-12-07T23:48:02.367 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 57 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:02.367 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:48:02.367 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:48:02.367 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:02.367 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 10 client.4355.objecter _op_submit oid noexist '@15' '@15' [read 0~0] tid 15 osd.1 2023-12-07T23:48:02.367 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 15 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 15 client.4355.objecter _send_op 15 to 15.1 on osd.1 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:15 15.1 15:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x55c9f7c44b50 con 0x55c9f7b62860 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:01.245+0000 7f654320c780 5 client.4355.objecter 2 in flight 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 21 ==== osd_map(58..58 src has 1..58) v4 ==== 636+0+0 (secure 0 0 0) 0x7f653405ee20 con 0x55c9f7adff80 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(58..58 src has 1..58) v4 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [58,58] > 57 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 58 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 10 client.4355.objecter checking op 14 2023-12-07T23:48:02.368 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 58 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:02.369 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:02.369 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 10 client.4355.objecter checking op 15 2023-12-07T23:48:02.369 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 58 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:02.369 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:48:02.369 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.363+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:48:02.369 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.364+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:48:02.369 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.364+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:48:02.755 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:02.621+0000 ea50640 -1 osd.4 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:03.068 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:03.063+0000 ea50640 -1 osd.2 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:03.890 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:03.867+0000 ea50640 -1 osd.4 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:04.265 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:04.155+0000 ea50640 -1 osd.2 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.364+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=59}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.364+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.364+0000 7f653d898640 20 client.4355.objecter 14 15.d3a59201 osd.1 exist [read 0~0] 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:02.364+0000 7f653d898640 20 client.4355.objecter 15 15.a7bbaa1 osd.1 noexist [read 0~0] 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.122+0000 7f653c896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 22 ==== osd_map(59..59 src has 1..59) v4 ==== 682+0+0 (secure 0 0 0) 0x7f653404d060 con 0x55c9f7adff80 2023-12-07T23:48:04.551 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(59..59 src has 1..59) v4 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [59,59] > 58 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 59 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 10 client.4355.objecter checking op 14 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 59 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 10 client.4355.objecter checking op 15 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 59 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 10 monclient: _renew_subs 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=60}) v3 -- 0x7f652c081650 con 0x55c9f7adff80 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:48:04.552 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 20 client.4355.objecter 14 15.d3a59201 osd.1 exist [read 0~0] 2023-12-07T23:48:04.553 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:03.496+0000 7f653d898640 20 client.4355.objecter 15 15.a7bbaa1 osd.1 noexist [read 0~0] 2023-12-07T23:48:04.553 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:04.553 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:04.553 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:04.553 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== mon.2 v2:172.21.15.129:3301/0 23 ==== osd_map(60..60 src has 1..60) v4 ==== 1000+0+0 (secure 0 0 0) 0x7f653405ee20 con 0x55c9f7adff80 2023-12-07T23:48:04.867 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:04.860+0000 ea50640 -1 osd.4 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(60..60 src has 1..60) v4 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [60,60] > 59 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 60 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 10 client.4355.objecter checking op 14 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 60 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 10 client.4355.objecter checking op 15 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 60 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:05.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 20 client.4355.objecter 14 15.d3a59201 osd.1 exist [read 0~0] 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:04.547+0000 7f653d898640 20 client.4355.objecter 15 15.a7bbaa1 osd.1 noexist [read 0~0] 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.312+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 17 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 14 ondisk uv 4 in 15.1 attempt 0 2023-12-07T23:48:05.621 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 5 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 14 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 15 client.4355.objecter _finish_op 14 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 14 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 5 client.4355.objecter 1 in flight 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 18 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 15 ondisk uv 0 in 15.1 attempt 0 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 15 2023-12-07T23:48:05.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 15 client.4355.objecter _finish_op 15 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (4371 ms) 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 15 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c21890 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 60 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [read 0~0] tid 16 osd.1 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:05.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 16 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 15 client.4355.objecter _send_op 16 to 15.1 on osd.1 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:16 15.1 15:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) v8 -- 0x55c9f7c23f70 con 0x55c9f7b62860 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c44590 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 60 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 10 client.4355.objecter _op_submit oid noexist '@15' '@15' [read 0~0] tid 17 osd.1 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:05.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 17 2023-12-07T23:48:05.659 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:05.599+0000 ea50640 -1 osd.2 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:06.185 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:06.176+0000 ea50640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:06.767 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:06.712+0000 ea50640 -1 osd.2 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:07.270 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:07.254+0000 ea50640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 15 client.4355.objecter _send_op 17 to 15.1 on osd.1 2023-12-07T23:48:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:17 15.1 15:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) v8 -- 0x55c9f7c64d80 con 0x55c9f7b62860 2023-12-07T23:48:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:48:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:05.616+0000 7f654320c780 5 client.4355.objecter 2 in flight 2023-12-07T23:48:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:06.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:06.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:06.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.122+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.122+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.122+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 19 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 16 ondisk uv 4 in 15.1 attempt 0 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 5 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 16 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 15 client.4355.objecter _finish_op 16 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 5 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 16 2023-12-07T23:48:07.789 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 5 client.4355.objecter 1 in flight 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 20 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 17 ondisk uv 0 in 15.1 attempt 0 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 10 client.4355.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 17 2023-12-07T23:48:07.790 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 15 client.4355.objecter _finish_op 17 2023-12-07T23:48:08.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 20 client.435 api_misc: [==========] Running 12 tests from 4 test suites. 2023-12-07T23:48:08.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] Global test environment set-up. 2023-12-07T23:48:08.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-12-07T23:48:08.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-12-07T23:48:08.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-12-07T23:48:08.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-07T23:48:08.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2023-12-07T23:47:10.563+0000 7f411223a780 -1 failed for service _ceph-mon._tcp 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2023-12-07T23:47:10.563+0000 7f411223a780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (458 ms) 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2008 ms) 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2466 ms total) 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e7606f70 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e69fbd40 2023-12-07T23:48:08.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e69f7ca0 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d80130 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d80b00 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d814d0 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d81eb0 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d82870 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76eebf0 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6cd4af0 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d76b60 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f0000 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76eea00 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f0f40 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76efdb0 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f1e30 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f0cb0 2023-12-07T23:48:08.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f29b0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d833b0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d83e40 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f1bf0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d84cd0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d83c10 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d85c20 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d84aa0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d86ac0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d85960 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d87930 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d86860 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d88470 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d88ed0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b5e560 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d877e0 2023-12-07T23:48:08.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b7fae0 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d88c80 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b80620 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b81060 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b7f850 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b81ed0 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b80e30 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b82d40 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b81c50 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b83b80 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b82b40 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b84a20 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b83930 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b85880 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b84810 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b86740 2023-12-07T23:48:08.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b85660 2023-12-07T23:48:08.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b87620 2023-12-07T23:48:08.808 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:08.697+0000 ea50640 -1 osd.4 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:09.164 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:09.157+0000 ea50640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:09.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: starte api_pool: Running main() from gmock_main.cc 2023-12-07T23:48:09.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] Global test environment set-up. 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolList (6649 ms) 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5913 ms) 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (4558 ms) 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4343 ms) 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-12-07T23:48:09.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5689 ms) 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (11138 ms) 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5827 ms) 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6135 ms) 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9108 ms) 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] 9 tests from LibRadosPools (59362 ms total) 2023-12-07T23:48:09.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: 2023-12-07T23:48:09.480 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] Global test environment tear-down 2023-12-07T23:48:09.480 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (59362 ms total) 2023-12-07T23:48:09.480 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ PASSED ] 9 tests. 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout:5.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 17 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (2169 ms) 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 20 librados: queue_aio_write 0x55c9f7a51480 completion 0x55c9f7c21780 write_seq 4 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c44590 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 60 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:48:09.492 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 18 osd.1 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 18 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 15 client.4355.objecter _send_op 18 to 15.1 on osd.1 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:18 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) v8 -- 0x55c9f7c23f70 con 0x55c9f7b62860 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:07.785+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:08.123+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:08.123+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:08.123+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.123+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.123+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:09.493 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.123+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:09.494 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 21 ==== osd_op_reply(18 exist [writefull 0~5] v60'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:09.494 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(18 exist [writefull 0~5] v60'5 uv5 ondisk = 0) v8 2023-12-07T23:48:09.494 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:09.494 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 18 ondisk uv 5 in 15.1 attempt 0 2023-12-07T23:48:09.494 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:48:09.494 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 18 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 15 client.4355.objecter _finish_op 18 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 18 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.488+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654089e640 20 librados: complete_aio_write 0x55c9f7c21780 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1704 ms) 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 20 librados: queue_aio_write 0x55c9f7a51480 completion 0x55c9f7c64c70 write_seq 5 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c21890 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 60 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:48:09.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 19 osd.1 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 19 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 15 client.4355.objecter _send_op 19 to 15.1 on osd.1 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:19 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) v8 -- 0x55c9f7c23f70 con 0x55c9f7b62860 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.489+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 22 ==== osd_map(61..62 src has 1..62) v4 ==== 2676+0+0 (crc 0 0 0) 0x7f6538002e10 con 0x55c9f7b62860 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(61..62 src has 1..62) v4 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [61,62] > 60 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 61 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 10 client.4355.objecter checking op 19 2023-12-07T23:48:09.535 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 61 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:48:09.536 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:09.536 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 62 2023-12-07T23:48:09.536 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 10 client.4355.objecter checking op 19 2023-12-07T23:48:09.759 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:09.753+0000 ea50640 -1 osd.4 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:10.459 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:10.412+0000 ea50640 -1 osd.0 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 62 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:09.530+0000 7f653d898640 20 client.4355.objecter 19 15.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:10.123+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:10.123+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:10.123+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:10.313+0000 7f6542301640 10 client.4355.objecter tick 2023-12-07T23:48:11.135 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.123+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.123+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.123+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 23 ==== osd_op_reply(19 exist [writefull 0~5] v62'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(19 exist [writefull 0~5] v62'6 uv6 ondisk = 0) v8 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 19 ondisk uv 6 in 15.1 attempt 0 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 19 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 15 client.4355.objecter _finish_op 19 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 19 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654089e640 20 librados: complete_aio_write 0x55c9f7c64c70 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1642 ms) 2023-12-07T23:48:11.136 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 20 librados: queue_aio_write 0x55c9f7a51480 completion 0x55c9f7c23f70 write_seq 6 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 10 client.4355.objecter _op_submit op 0x55c9f7c21890 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 20 client.4355.objecter _calc_target epoch 62 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 10 client.4355.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [1,6] primary 1 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 20 client.4355.objecter _get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 10 client.4355.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 20 osd.1 2023-12-07T23:48:11.137 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 20 client.4355.objecter get_session s=0x55c9f7a16140 osd=1 3 2023-12-07T23:48:11.291 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:11.175+0000 ea50640 -1 osd.4 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:11.677 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:11.637+0000 ea50640 -1 osd.0 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:12.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 15 client.4355.objecter _session_op_assign 1 20 2023-12-07T23:48:12.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 15 client.4355.objecter _send_op 20 to 15.1 on osd.1 2023-12-07T23:48:12.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- osd_op(unknown.0.0:20 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62) v8 -- 0x55c9f7c44730 con 0x55c9f7b62860 2023-12-07T23:48:12.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:12.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.131+0000 7f654320c780 5 client.4355.objecter 1 in flight 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 24 ==== osd_map(63..64 src has 1..65) v4 ==== 943+0+0 (crc 0 0 0) 0x7f6538002e10 con 0x55c9f7b62860 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_map(63..64 src has 1..65) v4 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 3 client.4355.objecter handle_osd_map got epochs [63,64] > 62 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 63 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 10 client.4355.objecter checking op 20 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 63 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 3 client.4355.objecter handle_osd_map decoding incremental epoch 64 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 10 client.4355.objecter checking op 20 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 20 client.4355.objecter _calc_target epoch 64 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 20 client.4355.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 20 client.4355.objecter dump_active .. 0 homeless 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:11.514+0000 7f653d898640 20 client.4355.objecter 20 15.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.123+0000 7f653c896640 10 monclient: tick 2023-12-07T23:48:12.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.123+0000 7f653c896640 10 monclient: _check_auth_tickets 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.123+0000 7f653c896640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 <== osd.1 v2:172.21.15.129:6816/3964479910 25 ==== osd_op_reply(20 exist [writefull 0~5] v64'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6538002880 con 0x55c9f7b62860 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 10 client.4355.objecter ms_dispatch 0x55c9f7a4b1b0 osd_op_reply(20 exist [writefull 0~5] v64'7 uv7 ondisk = 0) v8 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 10 client.4355.objecter in handle_osd_op_reply 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 7 client.4355.objecter handle_osd_op_reply 20 ondisk uv 7 in 15.1 attempt 0 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 10 client.4355.objecter op 0 rval 0 len 0 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 15 client.4355.objecter handle_osd_op_reply completed tid 20 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 15 client.4355.objecter _finish_op 20 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 4 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 15 client.4355.objecter _session_op_remove 1 20 2023-12-07T23:48:12.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 5 client.4355.objecter 0 in flight 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 20 librados: complete_aio_write 0x55c9f7c23f70 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1338 ms) 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 10 librados: watch_flush enter 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 10 librados: watch_flush exit 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 10 client.4355.objecter close_session for osd.1 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x55c9f7b62860 msgr2=0x55c9f7c205a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x55c9f7b62860 0x55c9f7c205a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 20 client.4355.objecter put_session s=0x55c9f7a16140 osd=1 1 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 20 client.4355.objecter shutdown clearing up homeless session... 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 10 client.4355.objecter successfully canceled tick 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f652c06aac0 msgr2=0x7f652c06cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f652c06aac0 0x7f652c06cf80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55c9f7b7db60 tx=0x7f653000c040 comp rx=0 tx=0).stop 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 10 monclient: shutdown 2023-12-07T23:48:12.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7adff80 msgr2=0x55c9f7b6c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 --2- 172.21.15.129:0/1088732227 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c9f7adff80 0x55c9f7b6c800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f653400e670 tx=0x7f653400eb30 comp rx=0 tx=0).stop 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654089e640 1 -- 172.21.15.129:0/1088732227 reap_dead start 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 shutdown_connections 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 >> 172.21.15.129:0/1088732227 conn(0x55c9f7ae2180 msgr2=0x55c9f7ae0360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 shutdown_connections 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 -- 172.21.15.129:0/1088732227 wait complete. 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.470+0000 7f654320c780 1 librados: shutdown 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] 12 tests from AsioRados (31422 ms total) 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2023-12-07T23:48:12.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] Global test environment tear-down 2023-12-07T23:48:12.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (62383 ms total) 2023-12-07T23:48:12.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ PASSED ] 12 tests. 2023-12-07T23:48:12.525 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:12.379+0000 ea50640 -1 osd.2 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4466.0:2 41.6 41:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T23:48:12.857 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:12.791+0000 ea50640 -1 osd.4 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:12.969 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:12.922+0000 ea50640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:14.069 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:13.982+0000 ea50640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (63890 ms) 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (63890 ms total) 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (63890 ms total) 2023-12-07T23:48:14.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-12-07T23:48:14.218 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:14.212+0000 ea50640 -1 osd.4 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:14.553 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:14.547+0000 ea50640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16.77fd7b1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:15.347 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:15.244+0000 ea50640 -1 osd.4 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:15.470 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:15.291+0000 ea50640 -1 osd.0 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:15.650 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:12.122 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.658 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:12.130 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.659 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:12.132 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.663 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:12.136 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.667 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:12.112 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.672 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:12.117 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.894 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:12.366 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.911 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:12.383 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:15.977 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:12.422 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:16.578 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:16.502+0000 ea50640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:16.598 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:16.522+0000 ea50640 -1 osd.4 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:17.110 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:16.930+0000 ea50640 -1 osd.2 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:17.694 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:17.676+0000 ea50640 -1 osd.4 67 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:17.986 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:48:17.976+0000 ea50640 -1 osd.1 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4572.0:2 58.1b 58.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:18.052 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:17.858+0000 ea50640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:18.524 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:18.508+0000 ea50640 -1 osd.2 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:18.839 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:18.834+0000 ea50640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:18.851 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:18.806+0000 ea50640 -1 osd.4 68 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:18.984 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:48:18.969+0000 ea50640 -1 osd.1 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4572.0:2 58.1b 58.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:19.962 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:19.877+0000 ea50640 -1 osd.4 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:20.167 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:48:20.162+0000 ea50640 -1 osd.1 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4572.0:2 58.1b 58:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:20.209 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:20.135+0000 ea50640 -1 osd.2 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:20.446 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:20.440+0000 ea50640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:21.426 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:21.421+0000 ea50640 -1 osd.2 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:22.230 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:22.163+0000 ea50640 -1 osd.4 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4606.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2023-12-07T23:48:22.269 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:22.250+0000 ea50640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:22.791 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:22.678+0000 ea50640 -1 osd.2 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:23.263 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:23.257+0000 ea50640 -1 osd.0 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:23.546 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:48:23.522+0000 ea50640 -1 osd.1 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4572.0:2 58.1b 58:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:23.662 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:23.622+0000 ea50640 -1 osd.4 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4418.0:4 31.16 31:681f27cd:::2.obj:head [write 0~179 in=179b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2023-12-07T23:48:23.906 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:23.815+0000 ea50640 -1 osd.2 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:24.442 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:24.370+0000 ea50640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:24.704 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-07T23:48:24.621+0000 ea50640 -1 osd.1 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4572.0:2 58.1b 58:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:24.733 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-07T23:48:24.707+0000 ea50640 -1 osd.4 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4418.0:4 31.16 31:681f27cd:::2.obj:head [write 0~179 in=179b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2023-12-07T23:48:24.945 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:24.928+0000 ea50640 -1 osd.2 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:25.496 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:25.457+0000 ea50640 -1 osd.0 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:26.151 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-07T23:48:26.101+0000 ea50640 -1 osd.2 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:5 16.11 16:8debfee0:test-rados-api-smithi129-92867-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-07T23:48:26.678 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:26.673+0000 ea50640 -1 osd.0 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:24 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2023-12-07T23:48:36.513 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:32.985 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.520 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:32.992 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.575 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:33.046 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.576 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:33.020 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.584 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:33.029 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.585 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:33.058 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.691 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:33.163 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.713 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:33.158 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:36.735 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:33.208 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:38.717 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:38.580+0000 ea50640 -1 osd.0 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4625.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2023-12-07T23:48:40.060 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:40.045+0000 ea50640 -1 osd.0 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4625.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2023-12-07T23:48:41.153 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:41.124+0000 ea50640 -1 osd.0 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4625.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2023-12-07T23:48:42.347 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:42.338+0000 ea50640 -1 osd.0 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4625.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2023-12-07T23:48:42.365 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:48:42.358+0000 ea50640 -1 osd.3 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4637.0:3 67.6 67:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) 2023-12-07T23:48:43.526 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:48:43.465+0000 ea50640 -1 osd.3 93 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4637.0:3 67.6 67:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) 2023-12-07T23:48:43.773 INFO:tasks.workunit.client.0.smithi129.stdout:d 0x5629e6b864f0 2023-12-07T23:48:43.773 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b884c0 2023-12-07T23:48:43.773 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b873b0 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b89360 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b88240 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8a1d0 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b89110 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8b090 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b89fb0 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8bf20 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8ae60 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8ca90 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8d500 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8bce0 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8e390 2023-12-07T23:48:43.774 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8d300 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8f2c0 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8e150 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6b8ef40 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d89e60 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d83540 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f0dc0 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d83220 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d865d0 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d85680 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d847f0 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f2740 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f0930 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f1870 2023-12-07T23:48:43.775 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76ef580 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d7d3d0 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d7ddd0 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d7e7d0 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d8a830 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e6d8b230 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5629e76f19c0 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 87 aios 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e7606f70 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e69fbd40 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e69f7ca0 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d80130 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d80b00 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d814d0 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d81eb0 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d82870 2023-12-07T23:48:43.776 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76eebf0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6cd4af0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d76b60 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f0000 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76eea00 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f0f40 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76efdb0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f1e30 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f0cb0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f29b0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d833b0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d83e40 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f1bf0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d84cd0 2023-12-07T23:48:43.777 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d83c10 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d85c20 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d84aa0 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d86ac0 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d85960 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d87930 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d86860 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d88470 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d88ed0 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b5e560 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d877e0 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b7fae0 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d88c80 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b80620 2023-12-07T23:48:43.778 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b81060 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b7f850 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b81ed0 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b80e30 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b82d40 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b81c50 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b83b80 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b82b40 2023-12-07T23:48:43.779 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b84a20 2023-12-07T23:48:44.527 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-07T23:48:44.521+0000 ea50640 -1 osd.3 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4637.0:3 67.6 67:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) 2023-12-07T23:48:45.377 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:45.359+0000 ea50640 -1 osd.0 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4625.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2023-12-07T23:48:46.609 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-07T23:48:46.566+0000 ea50640 -1 osd.0 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4625.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2023-12-07T23:48:51.290 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:48:51.125+0000 ea50640 -1 osd.5 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4858.0:2 77.6 77:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73) 2023-12-07T23:48:52.226 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:48:52.207+0000 ea50640 -1 osd.5 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4858.0:2 77.6 77:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73) 2023-12-07T23:48:55.942 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:52.415 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:55.946 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:52.419 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:55.947 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:52.419 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:56.012 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:52.457 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:56.019 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:52.464 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:56.025 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:52.497 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:56.113 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:03:52.586 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:56.141 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:03:52.586 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:56.148 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:03:52.621 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:48:57.823 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:48:57.819+0000 ea50640 -1 osd.5 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 79.15 79:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2023-12-07T23:49:01.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting cls: Running main() from gmock_main.cc 2023-12-07T23:49:01.196 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [==========] Running 2 tests from 1 test suite. 2023-12-07T23:49:01.196 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] Global test environment set-up. 2023-12-07T23:49:01.196 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] 2 tests from NeoRadosCls 2023-12-07T23:49:01.196 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ RUN ] NeoRadosCls.DNE 2023-12-07T23:49:01.196 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ OK ] NeoRadosCls.DNE (32590 ms) 2023-12-07T23:49:01.197 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2023-12-07T23:49:01.197 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (78271 ms) 2023-12-07T23:49:01.197 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] 2 tests from NeoRadosCls (110861 ms total) 2023-12-07T23:49:01.197 INFO:tasks.workunit.client.0.smithi129.stdout: cls: 2023-12-07T23:49:01.197 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] Global test environment tear-down 2023-12-07T23:49:01.197 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [==========] 2 tests from 1 test suite ran. (110861 ms total) 2023-12-07T23:49:01.197 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ PASSED ] 2 tests. 2023-12-07T23:49:22.416 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-07T23:49:22.358+0000 ea50640 -1 osd.5 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 84.0s0 84:1d13e705:test-rados-api-smithi129-92940-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2023-12-07T23:49:37.740 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:49:37.730+0000 ea50640 -1 osd.6 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4585.0:21 24.16 24:6df9c5af:::19.obj:head [write 0~152 in=152b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) 2023-12-07T23:49:38.787 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-07T23:49:38.778+0000 ea50640 -1 osd.6 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4585.0:21 24.16 24:6df9c5af:::19.obj:head [write 0~152 in=152b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) 2023-12-07T23:49:41.701 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:04:38.173 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:41.769 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:04:38.241 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:41.772 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:04:38.217 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:41.783 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:04:38.228 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:41.834 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:04:38.306 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:41.837 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:04:38.309 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:42.061 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:04:38.506 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:42.149 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:04:38.621 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:42.288 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:04:38.760 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:49:54.278 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: Running main() from gmock_main.cc 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] Global test environment set-up. 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6358 ms) 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (13148 ms) 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (144375 ms) 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] 3 tests from NeoRadosCmd (163881 ms total) 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: 2023-12-07T23:49:54.279 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] Global test environment tear-down 2023-12-07T23:49:54.280 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [==========] 3 tests from 1 test suite ran. (163881 ms total) 2023-12-07T23:49:54.280 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ PASSED ] 3 tests. 2023-12-07T23:49:54.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (1184 ms) 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (160 ms) 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (162895 ms) 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (164239 ms total) 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: 2023-12-07T23:49:54.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-12-07T23:49:54.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (164239 ms total) 2023-12-07T23:49:54.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] Global test environment set-up. 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 7 tests from LibRadosList 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjects (125584 ms) 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (29656 ms) 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo2,foo3 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2023-12-07T23:50:06.474 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo2 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo3 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo4,foo5 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo4 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo5 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo6,foo7 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo7 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo6 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo4 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo5 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo7 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo6 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo1 2023-12-07T23:50:06.475 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo1 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo2 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo3 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (12653 ms) 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 1 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 10 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 13 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 7 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 14 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 0 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 15 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 11 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 5 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 8 0 2023-12-07T23:50:06.476 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 6 0 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 3 0 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 4 0 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 12 0 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 9 0 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2 0 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (2220 ms) 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: x cursor=MIN 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2023-12-07T23:50:06.477 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: FIRST> seek to MIN oid=1 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2023-12-07T23:50:06.478 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:02547ec2:::1:head 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:02547ec2:::1:head -> 1 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:52ea6a34:::10:head 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:566253c9:::13:head 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:566253c9:::13:head -> 13 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2023-12-07T23:50:06.479 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2023-12-07T23:50:10.640 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=14 cursor=7:62a1935d:::14:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:62a1935d:::14:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:62a1935d:::14:head -> 14 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:6cac518f:::0:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:6cac518f:::0:head -> 0 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:863748b0:::15:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:863748b0:::15:head -> 15 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:89d3ae78:::11:head 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2023-12-07T23:50:10.641 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:b29083e3:::5:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:b29083e3:::5:head -> 5 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:cfc208b3:::3:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2023-12-07T23:50:10.642 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:d83876eb:::4:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:d83876eb:::4:head -> 4 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:e960b815:::9:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:e960b815:::9:head -> 9 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:f905c69b:::2:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:f905c69b:::2:head -> 2 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:b29083e3:::5:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:b29083e3:::5:head -> 5 2023-12-07T23:50:10.643 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=5 expected=5 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:89d3ae78:::11:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=11 expected=11 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:6cac518f:::0:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:6cac518f:::0:head -> 0 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=0 expected=0 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:62a1935d:::14:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:62a1935d:::14:head -> 14 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=14 expected=14 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2023-12-07T23:50:10.644 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=7 expected=7 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:566253c9:::13:head 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:566253c9:::13:head -> 13 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=13 expected=13 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:02547ec2:::1:head 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:02547ec2:::1:head -> 1 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=1 expected=1 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:e960b815:::9:head 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:e960b815:::9:head -> 9 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=9 expected=9 2023-12-07T23:50:10.645 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2023-12-07T23:50:16.995 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: Running main() from gmock_main.cc 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] Global test environment set-up. 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1009 ms) 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (79 ms) 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (159268 ms) 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:45.097233+0000 mon.c [INF] from='client.? 172.21.15.129:0/2356275713' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-92769-6", "tierpool": "test-rados-api-smithi129-92769-7", "force_nonempty": "--force-nonempty" }]: dispatch 2023-12-07T23:50:16.996 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:47.960909+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-92769-6", "tierpool": "test-rados-api-smithi129-92769-7", "force_nonempty": "--force-nonempty" }]: dispatch 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:49.287104+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 31 sec, osd.6 has slow ops) 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:52.263929+0000 mon.b [INF] from='client.? 172.21.15.129:0/2648320817' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:52.289267+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 26456.800ms) (OSD_SLOW_PING_TIME_BACK) 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:52.289935+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 22498.710ms) (OSD_SLOW_PING_TIME_FRONT) 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:53.022267+0000 client.admin [INF] onexx 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:53.047114+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:53.070406+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-92769-6", "tierpool": "test-rados-api-smithi129-92769-7", "force_nonempty": "--force-nonempty" }]': finished 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:57.660988+0000 mon.c [INF] from='client.? 172.21.15.129:0/1753429172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-92500-22"}]: dispatch 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:58.318310+0000 mon.c [INF] from='client.4706 172.21.15.129:0/1303986017' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-92940-7"}]: dispatch 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:49:59.703719+0000 mon.a [WRN] Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:00.003133+0000 mon.a [WRN] overall HEALTH_WARN Slow OSD heartbeats on back (longest 26456.800ms); Slow OSD heartbeats on front (longest 22498.710ms); 15 pool(s) do not have an application enabled 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:00.313958+0000 mon.a [INF] from='client.? 172.21.15.129:0/4275254991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-92488-16"}]: dispatch 2023-12-07T23:50:16.997 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:00.364075+0000 mon.b [INF] from='client.? 172.21.15.129:0/2648320817' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-07T23:50:17.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:00.887733+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"pr api_c_read_operations: Running main() from gmock_main.cc 2023-12-07T23:50:17.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-12-07T23:50:17.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (14625 ms) 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (58946 ms) 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (14734 ms) 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (64166 ms) 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (258 ms) 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-12-07T23:50:17.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (311 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (131 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (134 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (102 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (96 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (139 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (247 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (2805 ms) 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-12-07T23:50:17.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (6330 ms) 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (305 ms) 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (552 ms) 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (163882 ms total) 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (187118 ms total) 2023-12-07T23:50:17.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-12-07T23:50:18.709 INFO:tasks.workunit.client.0.smithi129.stdout:efix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-92500-22"}]: dispatch 2023-12-07T23:50:18.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:00.955549+0000 mon.a [INF] from='client.4706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-92940-7"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:02.053898+0000 mon.b [INF] from='client.? 172.21.15.129:0/2284287809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-92574-10"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:05.356444+0000 mon.b [INF] from='client.4775 172.21.15.129:0/2356275713' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-92769-6", "overlaypool": "test-rados-api-smithi129-92769-7"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:06.599628+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-92769-6", "overlaypool": "test-rados-api-smithi129-92769-7"}]': finished 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:06.953608+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:07.112772+0000 mon.a [INF] from='client.? 172.21.15.129:0/26328150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-92583-10"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:07.455720+0000 mon.a [INF] from='client.? 172.21.15.129:0/4275254991' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-92488-16"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:08.067704+0000 client.admin [INF] twoxx 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:08.670655+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:08.779714+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-92574-10"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:08.929035+0000 mon.a [INF] from='client.4775 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-92769-6", "overlaypool": "test-rados-api-smithi129-92769-7"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:07.919905+0000 mon.c [INF] from='client.? 172.21.15.129:0/1753429172' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-92500-22"}]: dispatch 2023-12-07T23:50:18.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:09.020062+0000 mon.b [INF] from='client.4626 172.21.15.129:0/2945228837' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-92867-7"}]: dispatch 2023-12-07T23:50:18.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:09.798574+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-92500-22"}]: dispatch 2023-12-07T23:50:18.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:10.895765+0000 mon.a [INF] from='client.4626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-92867-7"}]: dispatch 2023-12-07T23:50:18.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:10.960477+0000 mon.a [INF] from='client.? 172.21.15.129:0/1383923135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-93083-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-07T23:50:18.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:10.965024+0000 mon.a [INF] from='client.4706 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-92940-7"}]': finished 2023-12-07T23:50:22.562 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:11.418693+0000 mon.a [INF] from='client.? 172.21.15.129:0/26328150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-92583-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T23:50:22.562 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:11.612478+0000 mon.a [INF] from='client.? 172.21.15.129:0/4275254991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-92488-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T23:50:22.562 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:12.087754+0000 mon.a [WRN] Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-07T23:50:22.562 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:13.150358+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-12-07T23:50:22.562 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:12.021030+0000 mon.c [INF] from='client.? 172.21.15.129:0/1753429172' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-92500-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T23:50:22.562 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:12.182988+0000 mon.c [INF] from='client.4706 172.21.15.129:0/1303986017' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-92940-7"}]: dispatch 2023-12-07T23:50:22.562 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:12.682980+0000 mon.c [INF] from='client.? 172.21.15.129:0/3925182786' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-92467-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:12.980448+0000 mon.b [INF] from='client.? 172.21.15.129:0/2284287809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-92574-10"}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:13.465414+0000 mon.b [INF] from='client.? 172.21.15.129:0/3217989407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-92448-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:13.699504+0000 mon.c [INF] from='client.? 172.21.15.129:0/1903330217' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-92845-12"}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:14.605406+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-92500-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:14.708231+0000 mon.a [INF] from='client.4706 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-92940-7"}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:14.719606+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-92467-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:14.733956+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-92845-12"}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.156038+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-92574-10"}]: dispatch 2023-12-07T23:50:22.563 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.198209+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-92448-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.294973+0000 mon.a [INF] from='client.4626 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name" api_stat: Running main() from gmock_main.cc 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] Global test environment set-up. 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.Stat (27769 ms) 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (19173 ms) 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.StatNS (41356 ms) 2023-12-07T23:50:24.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (152 ms) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (142 ms) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 5 tests from LibRadosStat (88592 ms total) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (48056 ms) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (9305 ms) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (855 ms) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2821 ms) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (61037 ms total) 2023-12-07T23:50:24.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: 2023-12-07T23:50:24.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] Global test environment tear-down 2023-12-07T23:50:24.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (193844 ms total) 2023-12-07T23:50:24.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ PASSED ] 9 tests. 2023-12-07T23:50:26.969 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[93260]: starting. 2023-12-07T23:50:26.969 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[93260]: creating pool ceph_test_rados_list_parallel.smithi129-93202 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[93260]: created object 0... 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[93260]: created object 25... 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[93260]: created object 49... 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[93260]: finishing. 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[93260]: shutting down. 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[93261]: starting. 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[93261]: listing objects. 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[93261]: listed object 0... 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[93261]: listed object 25... 2023-12-07T23:50:26.970 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[93261]: saw 50 objects 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[93261]: shutting down. 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[110778]: starting. 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[110778]: creating pool ceph_test_rados_list_parallel.smithi129-93202 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[110778]: created object 0... 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[110778]: created object 25... 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[110778]: created object 49... 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[110778]: finishing. 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[110778]: shutting down. 2023-12-07T23:50:26.971 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[110779]: starting. 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[110779]: listing objects. 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[110779]: listed object 0... 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[110779]: listed object 25... 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[110779]: saw 41 objects 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[110779]: shutting down. 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[110780]: starting. 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[110780]: removed 25 objects... 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[110780]: removed half of the objects 2023-12-07T23:50:26.972 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[110780]: removed 50 objects... 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[110780]: removed 50 objects 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[110780]: shutting down. 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[117645]: starting. 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[117645]: creating pool ceph_test_rados_list_parallel.smithi129-93202 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[117645]: created object 0... 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[117645]: created object 25... 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[117645]: created object 49... 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[117645]: finishing. 2023-12-07T23:50:26.973 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[117645]: shutting down. 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[117646]: starting. 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[117646]: listing objects. 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[117646]: listed object 0... 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[117646]: listed object 25... 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[117646]: listed object 50... 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[117646]: saw 56 objects 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[117646]: shutting down. 2023-12-07T23:50:26.974 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.975 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.975 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[117647]: starting. 2023-12-07T23:50:26.975 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[117647]: added 25 objects... 2023-12-07T23:50:26.975 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[117647]: added half of the objects 2023-12-07T23:50:26.975 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[117647]: added 50 objects... 2023-12-07T23:50:26.975 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[117647]: added 50 objects 2023-12-07T23:50:26.976 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[117647]: shutting down. 2023-12-07T23:50:26.976 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:26.976 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: : "testprofile-LibRadosStatECPP_smithi129-92867-7"}]': finished 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.304589+0000 mon.a [INF] from='client.? 172.21.15.129:0/26328150' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-92583-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.308711+0000 mon.a [INF] from='client.? 172.21.15.129:0/4275254991' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-92488-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.364830+0000 mon.b [INF] from='client.4775 172.21.15.129:0/2356275713' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-92769-6"}]: dispatch 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.662356+0000 mon.a [INF] from='client.? 172.21.15.129:0/26328150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-92583-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-92583-10"}]: dispatch 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:15.899266+0000 mon.a [INF] from='client.? 172.21.15.129:0/4275254991' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-92488-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-92488-16"}]: dispatch 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:17.105817+0000 mon.a [WRN] Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:17.822617+0000 mon.a [INF] from='client.4775 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-92769-6"}]: dispatch 2023-12-07T23:50:28.015 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:16.784108+0000 mon.c [INF] from='client.? 172.21.15.129:0/1903330217' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-92845-12"}]: dispatch 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:16.843058+0000 mon.b [INF] from='client.? 172.21.15.129:0/2648320817' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:17.059531+0000 mon.b [INF] from='client.? 172.21.15.129:0/2284287809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-92574-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:17.232527+0000 mon.b [INF] from='client.4626 172.21.15.129:0/2945228837' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-92867-7"}]: dispatch 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:18.827206+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-92845-12"}]: dispatch 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:18.951027+0000 client.admin [INF] threexx 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:19.052857+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-92500-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:19.055447+0000 mon.a [INF] from='client.4706 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-92940-7"}]': finished 2023-12-07T23:50:28.016 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:19.057554+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-92467-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-07T23:50:28.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: api_stat_pp: Running main() from gmock_main.cc 2023-12-07T23:50:28.622 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: seed 92867 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (26786 ms) 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (72234 ms) 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (91 ms) 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-12-07T23:50:28.623 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (2753 ms) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (12444 ms) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (114308 ms total) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (34347 ms) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1595 ms) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1609 ms) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (2338 ms) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (39889 ms total) 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: 2023-12-07T23:50:28.624 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-12-07T23:50:28.625 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (198448 ms total) 2023-12-07T23:50:28.625 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-12-07T23:50:40.473 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.473 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[128256]: starting. 2023-12-07T23:50:40.473 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[128256]: creating pool ceph_test_rados_list_parallel.smithi129-93202 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[128256]: created object 0... 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[128256]: created object 25... 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[128256]: created object 49... 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[128256]: finishing. 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[128256]: shutting down. 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: starting. 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: listing objects. 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: listed object 0... 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: listed object 25... 2023-12-07T23:50:40.474 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: listed object 50... 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: listed object 75... 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: saw 100 objects 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[128257]: shutting down. 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[128264]: starting. 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[128264]: removed 25 objects... 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[128264]: removed half of the objects 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[128264]: removed 50 objects... 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[128264]: removed 50 objects 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[128264]: shutting down. 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.475 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[128260]: starting. 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[128260]: added 25 objects... 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[128260]: added half of the objects 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[128260]: added 50 objects... 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[128260]: added 50 objects 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[128260]: shutting down. 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[128259]: starting. 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[128259]: added 25 objects... 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[128259]: added half of the objects 2023-12-07T23:50:40.476 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[128259]: added 50 objects... 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[128259]: added 50 objects 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[128259]: shutting down. 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[137705]: starting. 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[137705]: creating pool ceph_test_rados_list_parallel.smithi129-93202 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[137705]: created object 0... 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[137705]: created object 25... 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[137705]: created object 49... 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[137705]: finishing. 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[137705]: shutting down. 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.477 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.478 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.478 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:40.478 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: starting. 2023-12-07T23:50:40.478 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: listing objects. 2023-12-07T23:50:41.451 INFO:tasks.ceph.mon.b.smithi184.stderr:2023-12-07T23:50:41.423+0000 ff7f640 -1 mon.b@1(peon).paxos(paxos updating c 1..501) lease_expire from mon.0 v2:172.21.15.129:3300/0 is 1.071606040s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2023-12-07T23:50:43.860 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]got: 2023-12-07T23:50:19.059682+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-92448-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-07T23:50:43.860 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:19.368836+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-07T23:50:43.861 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:19.395745+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-92574-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T23:50:43.861 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2023-12-07T23:50:19.425261+0000 mon.a [INF] from='client.4626 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-92867-7"}]: dispatch 2023-12-07T23:50:43.861 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (53298 ms) 2023-12-07T23:50:43.861 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (213654 ms total) 2023-12-07T23:50:43.861 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: 2023-12-07T23:50:43.862 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] Global test environment tear-down 2023-12-07T23:50:43.862 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (213654 ms total) 2023-12-07T23:50:43.862 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ PASSED ] 4 tests. 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[93308]: starting. 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[93308]: creating pool ceph_test_rados_open_pools_parallel.smithi129-93264 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[93308]: created object 0... 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[93308]: created object 25... 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[93308]: created object 49... 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[93308]: finishing. 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[93308]: shutting down. 2023-12-07T23:50:47.135 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[93310]: starting. 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[93310]: rados_pool_create. 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[93310]: rados_ioctx_create. 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[93310]: shutting down. 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[108868]: starting. 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[108868]: creating pool ceph_test_rados_open_pools_parallel.smithi129-93264 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[108868]: created object 0... 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[108868]: created object 25... 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[108868]: created object 49... 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[108868]: finishing. 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[108868]: shutting down. 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2023-12-07T23:50:47.136 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[108869]: starting. 2023-12-07T23:50:47.137 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[108869]: rados_pool_create. 2023-12-07T23:50:47.137 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[108869]: rados_ioctx_create. 2023-12-07T23:50:47.137 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[108869]: shutting down. 2023-12-07T23:50:47.137 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2023-12-07T23:50:47.137 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2023-12-07T23:50:47.137 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-12-07T23:50:51.803 INFO:tasks.workunit.client.0.smithi129.stdout:: listed object 0... 2023-12-07T23:50:51.803 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: listed object 25... 2023-12-07T23:50:51.803 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: listed object 50... 2023-12-07T23:50:51.803 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: listed object 75... 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: listed object 100... 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: listed object 125... 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: saw 150 objects 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[137706]: shutting down. 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[137707]: starting. 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[137707]: added 25 objects... 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[137707]: added half of the objects 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[137707]: added 50 objects... 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[137707]: added 50 objects 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[137707]: shutting down. 2023-12-07T23:50:51.804 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.805 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.805 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.805 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.805 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2023-12-07T23:50:51.805 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******* SUCCESS ********** 2023-12-07T23:50:53.501 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-12-07T23:50:53.501 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-12-07T23:50:53.501 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-12-07T23:50:53.501 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-12-07T23:50:53.501 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: seed 92583 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (23302 ms) 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (7786 ms) 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (36318 ms) 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (46853 ms) 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-12-07T23:50:53.502 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (29727 ms) 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4500 ms) 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4264 ms) 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (117 ms) 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (152867 ms total) 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (13949 ms) 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (325 ms) 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-12-07T23:50:53.503 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1167 ms) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1091 ms) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (358 ms) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (192 ms) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (261 ms) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (140 ms) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (17483 ms total) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (223455 ms total) 2023-12-07T23:50:53.504 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[93345]: starting. 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[93345]: creating pool ceph_test_rados_delete_pools_parallel.smithi129-93274 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[93345]: created object 0... 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[93345]: created object 25... 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[93345]: created object 49... 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[93345]: finishing. 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[93345]: shutting down. 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[93347]: starting. 2023-12-07T23:50:53.528 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[93347]: deleting pool ceph_test_rados_delete_pools_parallel.smithi129-93274 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[93347]: shutting down. 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[123255]: starting. 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[123255]: creating pool ceph_test_rados_delete_pools_parallel.smithi129-93274 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[123255]: created object 0... 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[123255]: created object 25... 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[123255]: created object 49... 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[123255]: finishing. 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[123255]: shutting down. 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[123257]: starting. 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[123257]: listing objects. 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[123257]: listed object 0... 2023-12-07T23:50:53.529 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[123257]: listed object 25... 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[123257]: saw 50 objects 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[123257]: shutting down. 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[123256]: starting. 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[123256]: deleting pool ceph_test_rados_delete_pools_parallel.smithi129-93274 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[123256]: shutting down. 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2023-12-07T23:50:53.530 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-12-07T23:50:53.567 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: Running main() from gmock_main.cc 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] Global test environment set-up. 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (22311 ms) 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (31870 ms) 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-12-07T23:50:53.568 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5921 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.Checksum (2690 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (57390 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (11846 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (18944 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1190 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.TruncTest (91 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (57 ms) 2023-12-07T23:50:53.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (93 ms) 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RmXattr (2398 ms) 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.XattrIter (711 ms) 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 14 tests from LibRadosIo (155513 ms total) 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (13740 ms) 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (441 ms) 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-12-07T23:50:53.570 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (379 ms) 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (191 ms) 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (549 ms) 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (196 ms) 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (279 ms) 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (212 ms) 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-12-07T23:50:53.571 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (575 ms) 2023-12-07T23:50:53.572 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-12-07T23:50:53.572 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (366 ms) 2023-12-07T23:50:53.572 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 10 tests from LibRadosIoEC (16928 ms total) 2023-12-07T23:50:53.572 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: 2023-12-07T23:50:53.572 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] Global test environment tear-down 2023-12-07T23:50:53.572 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [==========] 24 tests from 2 test suites ran. (223577 ms total) 2023-12-07T23:50:53.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ PASSED ] 24 tests. 2023-12-07T23:50:53.709 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:05:50.180 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:53.720 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:05:50.193 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:53.829 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:05:50.301 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:53.839 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:05:50.307 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:53.945 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:05:50.418 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:53.953 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:05:50.398 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:53.956 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:05:50.401 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:54.016 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:05:50.488 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:54.170 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:05:50.615 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:50:56.529 INFO:tasks.workunit.client.0.smithi129.stdout:0x5629e6b83930 2023-12-07T23:50:56.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b85880 2023-12-07T23:50:56.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b84810 2023-12-07T23:50:56.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b86740 2023-12-07T23:50:56.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b85660 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b87620 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b864f0 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b884c0 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b873b0 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b89360 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b88240 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8a1d0 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b89110 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8b090 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b89fb0 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8bf20 2023-12-07T23:50:56.530 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8ae60 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8ca90 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8d500 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8bce0 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8e390 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8d300 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8f2c0 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8e150 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6b8ef40 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d89e60 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d83540 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f0dc0 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d83220 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d865d0 2023-12-07T23:50:56.531 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d85680 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d847f0 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f2740 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f0930 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f1870 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76ef580 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d7d3d0 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d7ddd0 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d7e7d0 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d8a830 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e6d8b230 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5629e76f19c0 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: done. 2023-12-07T23:50:56.532 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (94603 ms) 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (94603 ms total) 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.Exec (15370 ms) 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (21019 ms) 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (10777 ms) 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-12-07T23:50:56.533 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4944 ms) 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (73022 ms) 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 8 tests from LibRadosMisc (125132 ms total) 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] Global test environment tear-down 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (226410 ms total) 2023-12-07T23:50:56.534 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ PASSED ] 12 tests. 2023-12-07T23:50:58.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-12-07T23:50:58.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-12-07T23:50:58.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-12-07T23:50:58.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-12-07T23:50:58.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify_test_cb 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (61912 ms) 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94840830994448 err -107 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (19502 ms) 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94840830994448 err -107 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (7176 ms) 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 403726925824 cookie 94840830994448 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (4153 ms) 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-12-07T23:50:58.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 446676598785 cookie 94840831014528 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (32875 ms) 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 566935683072 cookie 94840831039824 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (16472 ms) 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 618475290624 cookie 94840830981392 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 618475290624 cookie 94840831039824 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (9390 ms) 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 627065225216 cookie 94840830981392 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 627065225217 cookie 94840830981392 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (4298 ms) 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-12-07T23:50:58.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94840830981392 err -107 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4431 notify_id 644245094402 cookie 94840830981392 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (10429 ms) 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94840830981392 err -107 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1576 ms) 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (167783 ms total) 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify_test_cb 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (5135 ms) 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (5135 ms total) 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-12-07T23:50:58.692 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (228550 ms total) 2023-12-07T23:50:58.694 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-12-07T23:51:13.313 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:09.785 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.373 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:09.818 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.388 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:09.861 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.390 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:09.862 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.421 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:09.866 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.464 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:09.936 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.547 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:10.020 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.549 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:09.993 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:13.599 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:10.071 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:22.120 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: Running main() from gmock_main.cc 2023-12-07T23:51:22.120 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-12-07T23:51:22.120 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] Global test environment set-up. 2023-12-07T23:51:22.120 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-12-07T23:51:22.120 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: seed 92500 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (1 ms) 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (30367 ms) 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (7631 ms) 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (17358 ms) 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (10712 ms) 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3439 ms) 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (1893 ms) 2023-12-07T23:51:22.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2390 ms) 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (4849 ms) 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (13472 ms) 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (1170 ms) 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (3444 ms) 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-12-07T23:51:22.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3432 ms) 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (10977 ms) 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (37869 ms) 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (4316 ms) 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (1238 ms) 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (1130 ms) 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (592 ms) 2023-12-07T23:51:22.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (296 ms) 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (156578 ms total) 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (17529 ms) 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (19268 ms) 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5341 ms) 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (2163 ms) 2023-12-07T23:51:22.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-12-07T23:51:24.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibR api_lock: Running main() from gmock_main.cc 2023-12-07T23:51:24.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-12-07T23:51:24.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] Global test environment set-up. 2023-12-07T23:51:24.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-12-07T23:51:24.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-12-07T23:51:24.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (51097 ms) 2023-12-07T23:51:24.409 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockShared (69442 ms) 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (27772 ms) 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (3769 ms) 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3173 ms) 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.Unlock (794 ms) 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (208 ms) 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-12-07T23:51:24.410 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (160 ms) 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLock (156415 ms total) 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (9878 ms) 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (4742 ms) 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (7714 ms) 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (3728 ms) 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-12-07T23:51:24.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3905 ms) 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (593 ms) 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (3566 ms) 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (1247 ms) 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (35373 ms total) 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] Global test environment tear-down 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (254376 ms total) 2023-12-07T23:51:24.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ PASSED ] 16 tests. 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: Running main() from gmock_main.cc 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [==========] Running 12 tests from 1 test suite. 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] Global test environment set-up. 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] 12 tests from NeoRadosMisc 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Version 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Version (6293 ms) 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4579 ms) 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongName (96473 ms) 2023-12-07T23:51:28.347 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (15711 ms) 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (41122 ms) 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (35393 ms) 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Exec (25318 ms) 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (9339 ms) 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (12334 ms) 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.BigObject (5169 ms) 2023-12-07T23:51:28.348 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2208 ms) 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3962 ms) 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] 12 tests from NeoRadosMisc (257901 ms total) 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] Global test environment tear-down 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [==========] 12 tests from 1 test suite ran. (257901 ms total) 2023-12-07T23:51:28.349 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ PASSED ] 12 tests. 2023-12-07T23:51:33.282 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:29.755 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.286 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:29.758 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.360 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:29.804 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.361 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:29.806 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.364 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:29.837 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.367 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:29.839 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.381 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:29.853 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.465 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:29.937 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:33.494 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:29.938 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.562 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:50.034 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.628 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:50.100 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.690 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:50.134 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.698 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:50.143 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.746 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:06:50.218 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.752 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:50.224 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.758 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:50.230 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.788 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:06:50.233 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:51:53.840 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:06:50.312 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:00.004 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: seed 92765 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (17668 ms) 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (9149 ms) 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (19494 ms) 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (14758 ms) 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-12-07T23:52:00.005 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (10114 ms) 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (71183 ms total) 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (15931 ms) 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (12160 ms) 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (88584 ms) 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (32286 ms) 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (9943 ms) 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-07T23:52:00.006 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-83-gc56ad3ea/rpm/el9/BUILD/ceph-19.0.0-83-gc56ad3ea/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi129-92765-7 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: waiting for snaps to purge 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (21140 ms) 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (180045 ms total) 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (12593 ms) 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2194 ms) 2023-12-07T23:52:00.007 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-12-07T23:52:00.008 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (4359 ms) 2023-12-07T23:52:00.008 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-12-07T23:52:00.008 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1434 ms) 2023-12-07T23:52:00.008 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (20580 ms total) 2023-12-07T23:52:00.008 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2023-12-07T23:52:00.008 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN read_operations: Running main() from gmock_main.cc 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] Global test environment set-up. 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (25071 ms) 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (54078 ms) 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (7967 ms) 2023-12-07T23:52:03.215 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (11997 ms) 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (5793 ms) 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (59556 ms) 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (34381 ms) 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (35399 ms) 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (8047 ms) 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (16623 ms) 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2023-12-07T23:52:03.216 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (6026 ms) 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (15575 ms) 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (5724 ms) 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (6502 ms) 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (292739 ms total) 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] Global test environment tear-down 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (292739 ms total) 2023-12-07T23:52:03.217 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ PASSED ] 14 tests. 2023-12-07T23:52:08.006 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: Running main() from gmock_main.cc 2023-12-07T23:52:08.006 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2023-12-07T23:52:08.006 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] Global test environment set-up. 2023-12-07T23:52:08.006 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2023-12-07T23:52:08.006 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (39088 ms) 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (19985 ms) 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (13581 ms) 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (29949 ms) 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (21762 ms) 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (124365 ms total) 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (62011 ms) 2023-12-07T23:52:08.007 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (39632 ms) 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (34203 ms) 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (8931 ms) 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (7114 ms) 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi129-93569-11. 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: Waiting for snaps to purge. 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (21277 ms) 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (173168 ms total) 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] Global test environment tear-down 2023-12-07T23:52:08.008 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (297533 ms total) 2023-12-07T23:52:08.009 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ PASSED ] 11 tests. 2023-12-07T23:52:13.269 INFO:tasks.workunit.client.0.smithi129.stdout:adosIoECPP.RoundTripPP2 (784 ms) 2023-12-07T23:52:13.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-12-07T23:52:13.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (198 ms) 2023-12-07T23:52:13.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5606 ms) 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (1227 ms) 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (4723 ms) 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6232 ms) 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (6158 ms) 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3237 ms) 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8072 ms) 2023-12-07T23:52:13.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1148 ms) 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (7907 ms) 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1643 ms) 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (1658 ms) 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (92894 ms total) 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] Global test environment tear-down 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (303272 ms total) 2023-12-07T23:52:13.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-12-07T23:52:13.802 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:10.275 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:13.813 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:10.257 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:13.814 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:10.285 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:13.818 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:10.262 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:13.827 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:10.299 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:13.830 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:10.302 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:13.977 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:10.421 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:14.000 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:10.473 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:14.037 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:10.509 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: Running main() from gmock_main.cc 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: [==========] Running 16 tests from 2 test suites. 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] Global test environment set-up. 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] 14 tests from NeoRadosIo 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.Limits 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.Limits (36804 ms) 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2023-12-07T23:52:18.702 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (64545 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.ReadOp (21734 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.SparseRead (41157 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.RoundTrip (40274 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (29734 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (8096 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (5274 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (4144 ms) 2023-12-07T23:52:18.703 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.Trunc 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.Trunc (9578 ms) 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.Remove 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.Remove (5809 ms) 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3150 ms) 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.RmXattr (18191 ms) 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.GetXattrs (4298 ms) 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] 14 tests from NeoRadosIo (292789 ms total) 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] 2 tests from NeoRadosECIo 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosECIo.SimpleWrite 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosECIo.SimpleWrite (7910 ms) 2023-12-07T23:52:18.704 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosECIo.ReadOp 2023-12-07T23:52:18.705 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosECIo.ReadOp (7583 ms) 2023-12-07T23:52:18.705 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] 2 tests from NeoRadosECIo (15493 ms total) 2023-12-07T23:52:18.705 INFO:tasks.workunit.client.0.smithi129.stdout: io: 2023-12-07T23:52:18.705 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] Global test environment tear-down 2023-12-07T23:52:18.705 INFO:tasks.workunit.client.0.smithi129.stdout: io: [==========] 16 tests from 2 test suites ran. (308282 ms total) 2023-12-07T23:52:18.705 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ PASSED ] 16 tests. 2023-12-07T23:52:24.826 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: Running main() from gmock_main.cc 2023-12-07T23:52:24.826 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-12-07T23:52:24.826 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] Global test environment set-up. 2023-12-07T23:52:24.826 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-12-07T23:52:24.826 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (41556 ms) 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (19906 ms) 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (81723 ms) 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (17378 ms) 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (160563 ms total) 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (28644 ms) 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (11962 ms) 2023-12-07T23:52:24.827 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (10607 ms) 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (51214 ms total) 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (6885 ms) 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (6420 ms) 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (9156 ms) 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (7968 ms) 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (30429 ms total) 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2023-12-07T23:52:24.828 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (9895 ms) 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (7140 ms) 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (17036 ms total) 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] Global test environment tear-down 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (314667 ms total) 2023-12-07T23:52:24.829 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-12-07T23:52:27.089 INFO:tasks.workunit.client.0.smithi129.stdout: ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-12-07T23:52:27.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (9907 ms) 2023-12-07T23:52:27.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-12-07T23:52:27.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (8814 ms) 2023-12-07T23:52:27.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (5087 ms) 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (23808 ms total) 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (316981 ms total) 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-12-07T23:52:27.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-07T23:52:33.572 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:30.044 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.578 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:30.051 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.643 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:30.087 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.645 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:30.089 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.663 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:30.135 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.668 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:30.140 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.709 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:30.181 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.730 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:30.174 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:33.750 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:30.222 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.086 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:31.530 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.088 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:31.532 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.104 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:31.548 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.379 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:31.851 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.381 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:31.853 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.403 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:31.875 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.725 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:32.197 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.726 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:32.199 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:35.749 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:32.221 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:44.178 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: Running main() from gmock_main.cc 2023-12-07T23:52:44.178 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-12-07T23:52:44.178 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] Global test environment set-up. 2023-12-07T23:52:44.178 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5348 ms) 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (61463 ms) 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (37407 ms) 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (19287 ms) 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (16644 ms) 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (24580 ms) 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (40127 ms) 2023-12-07T23:52:44.179 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (28148 ms) 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (10295 ms) 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (22124 ms) 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (6422 ms) 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4956 ms) 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (4153 ms) 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-12-07T23:52:44.180 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3731 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.Flush (6376 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4668 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4321 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4852 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3125 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2769 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (4002 ms) 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-12-07T23:52:44.181 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4381 ms) 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3095 ms) 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4167 ms) 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3482 ms) 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4272 ms) 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 26 tests from LibRadosAio (334195 ms total) 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: 2023-12-07T23:52:44.182 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.Sim api_aio_pp: Running main() from gmock_main.cc 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6740 ms) 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (48019 ms) 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (72119 ms) 2023-12-07T23:52:48.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (13276 ms) 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (24499 ms) 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (40206 ms) 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (26982 ms) 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (8633 ms) 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (6547 ms) 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (5168 ms) 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-12-07T23:52:48.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (5338 ms) 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (8947 ms) 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (5369 ms) 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (6953 ms) 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (9024 ms) 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (5393 ms) 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (6830 ms) 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-12-07T23:52:48.252 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (5846 ms) 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3931 ms) 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3936 ms) 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3385 ms) 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3049 ms) 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3739 ms) 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4788 ms) 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-12-07T23:52:48.253 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3321 ms) 2023-12-07T23:52:48.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-12-07T23:52:48.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3072 ms) 2023-12-07T23:52:48.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-12-07T23:52:53.859 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:50.331 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:53.920 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:50.392 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:53.988 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:50.460 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:53.993 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:50.437 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:54.005 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:50.449 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:54.011 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:07:50.484 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:54.014 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:50.486 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:54.078 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:07:50.522 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:52:54.101 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:07:50.573 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.189 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:10.660 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.252 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:10.724 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.310 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:10.754 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.312 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:10.756 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.328 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:10.800 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.333 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:10.805 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.337 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:10.810 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.369 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:10.813 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:14.417 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:10.889 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.137 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:30.609 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.159 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:30.632 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.192 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:30.637 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.198 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:30.642 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.198 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:30.670 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.219 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:30.691 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.252 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:30.696 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.252 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:30.724 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:34.285 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:30.756 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.405 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:50.877 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.424 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:50.868 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.427 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:50.899 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.442 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:50.914 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.447 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:50.919 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.499 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:50.943 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.556 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:08:51.000 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.601 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:08:51.073 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:53:54.602 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:08:51.074 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRad api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=12 expected=12 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:d83876eb:::4:head 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:d83876eb:::4:head -> 4 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=4 expected=4 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:cfc208b3:::3:head 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2023-12-07T23:54:05.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=3 expected=3 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=6 expected=6 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=8 expected=8 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:863748b0:::15:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:863748b0:::15:head -> 15 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=15 expected=15 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:52ea6a34:::10:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2023-12-07T23:54:05.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=10 expected=10 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:f905c69b:::2:head 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:f905c69b:::2:head -> 2 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=2 expected=2 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (11036 ms) 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (165311 ms) 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (51670 ms) 2023-12-07T23:54:05.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 7 tests from LibRadosList (398131 ms total) 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2825 ms) 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo2,foo3 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo2 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo3 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo4,foo5 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo4 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo5 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo6,foo7 2023-12-07T23:54:05.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo7 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo6 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo4 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo5 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo7 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo6 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo1 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo1 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo2 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo3 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (4045 ms) 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 1 0 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 10 0 2023-12-07T23:54:05.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 13 0 2023-12-07T23:54:12.992 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 7 0 2023-12-07T23:54:12.992 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 14 0 2023-12-07T23:54:12.992 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 0 0 2023-12-07T23:54:12.992 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 15 0 2023-12-07T23:54:12.992 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 11 0 2023-12-07T23:54:12.992 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 5 0 2023-12-07T23:54:12.992 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 8 0 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 6 0 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 3 0 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 4 0 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 12 0 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 9 0 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2 0 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (4717 ms) 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 3 tests from LibRadosListEC (11587 ms total) 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3243 ms) 2023-12-07T23:54:12.993 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 1 test from LibRadosListNP (3243 ms total) 2023-12-07T23:54:12.994 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2023-12-07T23:54:12.994 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] Global test environment tear-down 2023-12-07T23:54:12.994 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [==========] 11 tests from 3 test suites ran. (422954 ms total) 2023-12-07T23:54:12.994 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ PASSED ] 11 tests. 2023-12-07T23:54:18.983 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:15.454 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:18.985 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:15.458 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:19.053 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:15.525 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:19.055 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:15.528 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:19.058 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:15.530 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:19.064 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:15.508 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:19.066 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:15.510 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:19.135 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:15.607 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:19.157 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:15.602 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.316 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:16.760 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.317 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:16.762 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.331 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:16.776 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.341 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:16.813 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.342 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:16.814 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.355 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:16.827 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.561 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:17.033 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.562 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:17.034 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:20.573 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:17.046 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:28.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-12-07T23:54:28.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-12-07T23:54:28.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-12-07T23:54:28.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-12-07T23:54:28.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-12-07T23:54:28.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (32404 ms) 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (41539 ms) 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (62516 ms) 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (24277 ms) 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (28813 ms) 2023-12-07T23:54:28.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (36686 ms) 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (211580 ms) 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (437815 ms total) 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (437816 ms total) 2023-12-07T23:54:28.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-12-07T23:54:30.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-12-07T23:54:30.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: seed 92659 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (184 ms) 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (51176 ms) 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (43285 ms) 2023-12-07T23:54:30.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (7273 ms) 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (26774 ms) 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3976 ms) 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (15159 ms) 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4976 ms) 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (42 ms) 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (262 ms) 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-12-07T23:54:30.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (386 ms) 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (1370 ms) 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (164 ms) 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: osd_max_attr_size = 0 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (183818 ms) 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (9548 ms) 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: done waiting, doing copies 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: waiting for final deep scrubs... 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: done waiting 2023-12-07T23:54:30.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (82348 ms) 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (371 ms) 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (171 ms) 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4294 ms) 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-12-07T23:54:30.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-12-07T23:54:30.580 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [-- write_operations: Running main() from gmock_main.cc 2023-12-07T23:54:30.580 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2023-12-07T23:54:30.580 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] Global test environment set-up. 2023-12-07T23:54:30.580 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2023-12-07T23:54:30.580 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2023-12-07T23:54:30.580 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (32406 ms) 2023-12-07T23:54:30.580 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (68864 ms) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (19484 ms) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (43304 ms) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (40598 ms) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (27843 ms) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (207589 ms) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (440089 ms total) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] Global test environment tear-down 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (440089 ms total) 2023-12-07T23:54:30.581 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ PASSED ] 7 tests. 2023-12-07T23:54:39.638 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:36.111 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.644 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:36.116 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.699 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:36.171 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.701 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:36.145 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.702 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:36.147 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.705 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:36.177 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.716 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:36.188 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.777 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:36.222 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:39.788 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:36.260 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:49.617 INFO:tasks.workunit.client.0.smithi129.stdout:pleWrite 2023-12-07T23:54:49.617 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8271 ms) 2023-12-07T23:54:49.617 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-12-07T23:54:49.617 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7936 ms) 2023-12-07T23:54:49.617 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7786 ms) 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (8246 ms) 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (9883 ms) 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (9377 ms) 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8317 ms) 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7897 ms) 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6956 ms) 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-12-07T23:54:49.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6610 ms) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7400 ms) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7331 ms) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7786 ms) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6858 ms) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7891 ms) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6891 ms) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (125436 ms total) 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: 2023-12-07T23:54:49.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] Global test environment tear-down 2023-12-07T23:54:49.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (459631 ms total) 2023-12-07T23:54:49.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ PASSED ] 42 tests. 2023-12-07T23:54:59.563 INFO:tasks.workunit.client.0.smithi129.stdout: list: Running main() from gmock_main.cc 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [==========] Running 6 tests from 2 test suites. 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] Global test environment set-up. 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] 3 tests from NeoradosList 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosList.ListObjects 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosList.ListObjects (145419 ms) 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosList.ListObjectsNS (41311 ms) 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosList.ListObjectsMany (193868 ms) 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] 3 tests from NeoradosList (380598 ms total) 2023-12-07T23:54:59.564 INFO:tasks.workunit.client.0.smithi129.stdout: list: 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] 3 tests from NeoradosECList 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosECList.ListObjects 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosECList.ListObjects (8262 ms) 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosECList.ListObjectsNS 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosECList.ListObjectsNS (13654 ms) 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosECList.ListObjectsMany 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosECList.ListObjectsMany (66615 ms) 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] 3 tests from NeoradosECList (88531 ms total) 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] Global test environment tear-down 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [==========] 6 tests from 2 test suites ran. (469129 ms total) 2023-12-07T23:54:59.565 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ PASSED ] 6 tests. 2023-12-07T23:54:59.785 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:56.257 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.818 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:56.263 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.825 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:56.270 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.835 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:56.308 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.890 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:09:56.335 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.891 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:56.363 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.901 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:56.373 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.962 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:56.435 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:54:59.988 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:09:56.459 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:01.891 INFO:tasks.workunit.client.0.smithi129.stdout:--------] 22 tests from LibRadosMiscPP (435577 ms total) 2023-12-07T23:55:01.891 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (3554 ms) 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (3554 ms total) 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (644 ms) 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (180 ms) 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (824 ms total) 2023-12-07T23:55:01.892 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (572 ms) 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (54 ms) 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (626 ms total) 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (676 ms) 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (178 ms) 2023-12-07T23:55:01.893 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (854 ms total) 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1726 ms) 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1726 ms total) 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (471788 ms total) 2023-12-07T23:55:01.894 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-12-07T23:55:19.683 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:16.155 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.743 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:16.215 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.792 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:16.265 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.798 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:16.243 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.802 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:16.247 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.812 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:16.285 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.816 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:16.288 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.857 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:16.301 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:19.866 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:16.339 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout:osAio.RoundTripAppendPP (3154 ms) 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3039 ms) 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3346 ms) 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3548 ms) 2023-12-07T23:55:33.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: setting pool EIO 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: max_success 0, min_failed 1 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5621 ms) 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (6414 ms) 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (360232 ms total) 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3725 ms) 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10039 ms) 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17633 ms) 2023-12-07T23:55:33.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (4731 ms) 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (36128 ms total) 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (4407 ms) 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (4407 ms total) 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (16906 ms) 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7795 ms) 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-12-07T23:55:33.144 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (8094 ms) 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7039 ms) 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3195 ms) 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7068 ms) 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7158 ms) 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7070 ms) 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (9123 ms) 2023-12-07T23:55:33.145 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-12-07T23:55:33.146 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7956 ms) 2023-12-07T23:55:33.146 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-12-07T23:55:33.146 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7130 ms) 2023-12-07T23:55:33.146 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-12-07T23:55:33.146 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6518 ms) 2023-12-07T23:55:33.146 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-12-07T23:55:33.146 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7336 ms) 2023-12-07T23:55:39.834 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:36.307 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.840 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:36.313 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.898 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:36.343 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.903 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:36.348 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.915 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:36.387 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.919 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:36.391 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.958 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:36.430 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.970 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:36.414 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:55:39.986 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:36.458 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.098 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:56.567 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.098 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:56.569 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.151 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:56.595 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.155 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:56.600 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.157 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:10:56.630 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.164 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:56.637 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.168 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:56.640 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.203 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:10:56.647 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:00.221 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:10:56.693 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.039 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:01.511 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.044 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:01.517 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.071 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:01.544 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.094 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:01.538 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.094 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:01.567 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.099 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:01.571 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.113 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:01.558 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.120 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:01.593 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:05.139 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:01.584 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.239 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:16.711 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.291 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:16.736 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.294 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:16.739 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.295 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:16.768 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.331 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:16.776 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.332 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:16.805 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.352 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:16.824 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.354 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:16.826 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:20.402 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:16.874 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:21.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3238 ms) 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7623 ms) 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7101 ms) 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7412 ms) 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7097 ms) 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7735 ms) 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7944 ms) 2023-12-07T23:56:21.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (150539 ms total) 2023-12-07T23:56:21.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2023-12-07T23:56:21.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-12-07T23:56:21.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (551306 ms total) 2023-12-07T23:56:21.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-12-07T23:56:40.325 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:36.798 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.327 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:36.800 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.363 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:36.836 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.375 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:36.819 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.375 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:36.847 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.376 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:36.821 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.376 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:36.849 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.417 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:36.861 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:56:40.418 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:36.891 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.283 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:56.756 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.285 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:56.758 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.329 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:56.774 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.329 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:56.775 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.336 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:56.808 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.337 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:56.810 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.344 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:11:56.817 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.359 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:11:56.804 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:00.368 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:11:56.840 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.298 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:11.771 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.339 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:11.812 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.346 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:11.791 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.349 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:11.795 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.363 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:11.835 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.365 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:11.838 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.366 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:11.838 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.367 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:11.812 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:15.383 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:11.855 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.471 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:21.943 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.475 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:21.948 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.523 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:21.968 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.528 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:21.973 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.534 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:22.007 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.536 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:22.008 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.553 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:21.998 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.566 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:22.038 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:25.579 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:22.052 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.480 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:26.953 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.482 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:26.927 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.485 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:26.957 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.488 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:26.933 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.507 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:26.980 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.512 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:26.957 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.535 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:27.008 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.542 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:27.015 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:30.561 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:27.033 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.551 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:42.023 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.555 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:42.028 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.588 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:42.061 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.612 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:42.085 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.615 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:42.060 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.616 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:42.089 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.618 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:42.063 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.648 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:42.120 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:45.648 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:42.093 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.547 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:42.992 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.548 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:42.993 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.557 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:43.029 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.557 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:43.002 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.557 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:43.030 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.566 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:43.039 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.633 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:43.106 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.634 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:43.107 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:57:46.643 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:43.116 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.693 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:57.166 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.696 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:57.169 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.726 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:57.199 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.746 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:57.218 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.747 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:57.220 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.750 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:57.195 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.752 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:57.197 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.777 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:12:57.250 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:00.780 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:12:57.225 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.724 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:02.197 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.742 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:02.215 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.761 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:02.234 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.774 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:02.219 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.775 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:02.220 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.779 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:02.251 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.780 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:02.253 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.792 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:02.237 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:05.795 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:02.268 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.750 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:12.223 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.753 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:12.225 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.753 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:12.197 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.755 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:12.200 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.782 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:12.227 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.785 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:12.258 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.807 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:12.279 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.809 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:12.282 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:15.837 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:12.310 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.536 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:12.981 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.536 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:12.982 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.546 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:12.991 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.589 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:13.061 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.589 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:13.062 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.598 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:13.071 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.621 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:13.094 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.622 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:13.095 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:16.630 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:13.103 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: seed 92769 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (30100 ms) 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (93093 ms) 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-12-07T23:58:17.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (552 ms) 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (123746 ms total) 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (87532 ms) 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (79435 ms) 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (22722 ms) 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [4] 2023-12-07T23:58:17.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [5,4] 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [6,5,4] 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting some heads 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 7 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 6 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 5 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 4 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrubs... 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (174108 ms) 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11580 ms) 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-12-07T23:58:17.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9851 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (9073 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10116 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11439 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10126 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (12270 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (39179 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10009 ms) 2023-12-07T23:58:17.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-12-07T23:58:17.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10073 ms) 2023-12-07T23:58:17.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-12-07T23:58:17.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16079 ms) 2023-12-07T23:58:17.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-12-07T23:58:17.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7853 ms) 2023-12-07T23:58:17.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-12-07T23:58:17.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11548 ms) 2023-12-07T23:58:25.794 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:22.267 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.798 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:22.271 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.798 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:22.244 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.804 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:22.249 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.828 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:22.301 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.837 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:22.282 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.867 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:22.338 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.869 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:22.342 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:25.892 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:22.365 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.699 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:27.172 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.703 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:27.176 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.720 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:27.193 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.750 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:27.195 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.751 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:27.197 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.752 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:27.224 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.758 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:27.229 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.772 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:27.244 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:30.772 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:27.218 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:40.042 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: ok, hit_set contains 278:602f83fe:::foo:head 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9815 ms) 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg_num = 32 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 0 ls 1701993514,0 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 1 ls 1701993514,0 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 2 ls 1701993514,0 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 3 ls 1701993515,0 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 4 ls 1701993514,0 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 5 ls 1701993514,0 2023-12-07T23:58:40.043 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 6 ls 1701993514,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 7 ls 1701993514,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 8 ls 1701993514,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 9 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 10 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 11 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 12 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 13 ls 1701993514,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 14 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 15 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 16 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 17 ls 1701993514,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 18 ls 1701993515,0 2023-12-07T23:58:40.044 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 19 ls 1701993514,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 20 ls 1701993514,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 21 ls 1701993515,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 22 ls 1701993514,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 23 ls 1701993514,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 24 ls 1701993515,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 25 ls 1701993514,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 26 ls 1701993515,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 27 ls 1701993515,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 28 ls 1701993514,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 29 ls 1701993515,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 30 ls 1701993514,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 31 ls 1701993515,0 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg_num = 32 2023-12-07T23:58:40.045 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6cac518f:::0:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:02547ec2:::1:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f905c69b:::2:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:cfc208b3:::3:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d83876eb:::4:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b29083e3:::5:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c4fdafeb:::6:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:5c6b0b28:::7:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:bd63b0f1:::8:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:e960b815:::9:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:52ea6a34:::10:head 2023-12-07T23:58:40.046 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:89d3ae78:::11:head 2023-12-07T23:58:40.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:de5d7c5f:::12:head 2023-12-07T23:58:40.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:566253c9:::13:head 2023-12-07T23:58:40.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:62a1935d:::14:head 2023-12-07T23:58:40.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:863748b0:::15:head 2023-12-07T23:58:40.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:3958e169:::16:head 2023-12-07T23:58:40.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:4d4dabf9:::17:head 2023-12-07T23:58:40.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:8391935d:::18:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:28883081:::19:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:69259c59:::20:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:4bdb80b7:::21:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a11c5d71:::22:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:271af37b:::23:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:95b121be:::24:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:58d1031b:::25:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:0a050783:::26:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c709704c:::27:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:cbe56eaf:::28:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:86b4b162:::29:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:70d89383:::30:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:dd450c7c:::31:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6d5729b1:::32:head 2023-12-07T23:58:40.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c388f3fb:::33:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:56cfea31:::34:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:9dbc1bf7:::35:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:40b74ccd:::36:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:4d5aaf42:::37:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:920f362c:::38:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6cc53222:::39:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:9cad833f:::40:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:1ea84d41:::41:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c4480ef6:::42:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a694361e:::43:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d1bd33e9:::44:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:ddc2cd5d:::45:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:2b782207:::46:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:7b187fca:::47:head 2023-12-07T23:58:40.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:90ecdf6f:::48:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a5ed95fe:::49:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:ea0eaa55:::50:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f33ef17b:::51:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a0d1b2f6:::52:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:60c5229e:::53:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:edcbc575:::54:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:102cf253:::55:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:efb7fb0b:::56:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:50d0a326:::57:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d4dc5daf:::58:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:3a130462:::59:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:ec87ed71:::60:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d5bc9454:::61:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:3ddfe313:::62:head 2023-12-07T23:58:40.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:7c2816b9:::63:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:47e00e4d:::64:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c6410c18:::65:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b48ed237:::66:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:cd63ad31:::67:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b179e92b:::68:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:0d9f741a:::69:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6d3352ae:::70:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c6d5c19e:::71:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:bc4729c3:::72:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:77e930b9:::73:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:0abeecfd:::74:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b7c37e15:::75:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b6378398:::76:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:02bd68de:::77:head 2023-12-07T23:58:40.052 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:cc795d2d:::78:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:630d4fea:::79:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:e0d29ef5:::80:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:fd6f13d2:::81:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:606461d5:::82:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:eadbdc43:::83:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:8761d0bb:::84:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:9ef0186f:::85:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:e0d41294:::86:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:961de695:::87:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:1423148f:::88:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:633a8fa2:::89:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a8653809:::90:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:3dac8b33:::91:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:35aad435:::92:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f6dcc343:::93:head 2023-12-07T23:58:40.053 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:dbbdad87:::94:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:1cb48ce0:::95:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:03cd461c:::96:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:17a4ea99:::97:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:9993c9a7:::98:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6394211c:::99:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:94c7ae57:::100:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6fdee5bb:::101:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:9a477fd1:::102:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:eb850916:::103:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:affc56b9:::104:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b42dc814:::105:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f319f8f0:::106:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:9a40b9de:::107:head 2023-12-07T23:58:40.054 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:8b524f28:::108:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:e3de589f:::109:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:90f90a5b:::110:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a7b4f1d7:::111:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:af51766e:::112:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b6f90bd1:::113:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:e0261208:::114:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c9569ef7:::115:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:61bebe50:::116:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:fe93412b:::117:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d3d38bee:::118:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:3100ba0c:::119:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d0560ada:::120:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f0ea8b35:::121:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:766f231a:::122:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a07a2582:::123:head 2023-12-07T23:58:40.055 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:bd7c6b3a:::124:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:fb2ddaff:::125:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:4408e1fe:::126:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:ee1df7a7:::127:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c3002909:::128:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:4f48ffa9:::129:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:edf38733:::130:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c08425c0:::131:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:5f902d98:::132:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:41ea2c93:::133:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:813cee13:::134:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:0131818d:::135:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:26ba5a85:::136:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:381b8a5a:::137:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:28797e47:::138:head 2023-12-07T23:58:40.056 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:bfca7f22:::139:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:36807075:::140:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:80b03975:::141:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:5c15709b:::142:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f39ea15e:::143:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:ea992956:::144:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:48887b1c:::145:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:9f24a9dd:::146:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:987f100b:::147:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d2dd3581:::148:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:7fed1808:::149:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c80b70e9:::150:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:85ed90f9:::151:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:36428b24:::152:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d044c34a:::153:head 2023-12-07T23:58:40.057 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:7c18bf58:::154:head 2023-12-07T23:58:40.058 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d1c21232:::155:head 2023-12-07T23:58:40.058 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a7a3c575:::156:head 2023-12-07T23:58:40.058 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:87da0633:::157:head 2023-12-07T23:58:40.058 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d5ac3822:::158:head 2023-12-07T23:58:40.058 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:3f20522d:::159:head 2023-12-07T23:58:40.058 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6ca26563:::160:head 2023-12-07T23:58:40.058 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:532ce135:::161:head 2023-12-07T23:58:40.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:c78863e6:::162:head 2023-12-07T23:58:45.846 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:42.318 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.873 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:42.346 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.895 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:42.368 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.902 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:42.347 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.903 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:42.348 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.904 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:42.377 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.905 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:42.378 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.924 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:42.370 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:45.929 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:42.401 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.771 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:47.244 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.774 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:47.247 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.788 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:13:47.261 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.824 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:47.297 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.825 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:47.270 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.826 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:47.298 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.829 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:47.274 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.838 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:13:47.311 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:58:50.842 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:13:47.287 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.808 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:02.281 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.811 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:02.285 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.838 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:02.311 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.856 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:02.301 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.859 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:02.304 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.863 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:02.336 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.864 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:02.337 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.881 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:02.326 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:05.890 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:02.362 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.580 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:03.025 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.581 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:03.053 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.581 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:03.026 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.582 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:03.054 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.588 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:03.034 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.596 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:03.069 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.601 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:03.074 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.602 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:03.075 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:06.611 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:03.084 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.096 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:22.569 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.097 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:22.570 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.131 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:22.604 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.138 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:22.583 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.142 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:22.615 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.143 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:22.589 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.145 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:22.618 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.178 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:22.624 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:26.181 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:22.654 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.121 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:32.594 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.127 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:32.601 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.155 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:32.628 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.171 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:32.616 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.174 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:32.619 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.176 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:32.649 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.183 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:32.655 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.198 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:32.643 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:36.207 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:32.680 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:45.974 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:42.447 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:45.978 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:42.451 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:46.001 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:42.474 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:46.022 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:42.467 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:46.025 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:42.470 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:46.037 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:42.482 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:46.052 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:42.525 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:46.053 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:42.526 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:46.070 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:42.543 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:7462ddf6:::163:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:5d165639:::164:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f43765fc:::165:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b4c720e9:::166:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:e694b040:::167:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:afa38db2:::168:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:77ba9f53:::169:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:87495034:::170:head 2023-12-07T23:59:50.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:7c96bf0e:::171:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:dbe346cc:::172:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:e943ec24:::173:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f97a9c0c:::174:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:6f26e74d:::175:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:4f95e106:::176:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:0e6f2f8f:::177:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:05db05f1:::178:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:38a78d66:::179:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:d095610b:::180:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:a1a9d709:::181:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:1e5d39db:::182:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:f7df4fb9:::183:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:03a7f161:::184:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:ba70721e:::185:head 2023-12-07T23:59:50.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:28e5662d:::186:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:973d52de:::187:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:4303eb1c:::188:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:b990b48e:::189:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:29b8165b:::190:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:3547f197:::191:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:7e260936:::192:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:1abec7b1:::193:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:10fdda93:::194:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:15817eea:::195:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:770bab57:::196:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:ed9e13e7:::197:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:71471a8f:::198:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 279:10fb1d02:::199:head 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22023 ms) 2023-12-07T23:59:50.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first is 1701993538 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,1701993543,1701993544,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,1701993543,1701993544,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,1701993543,1701993544,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,1701993543,1701993544,1701993546,1701993547,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,1701993543,1701993544,1701993546,1701993547,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993538,1701993540,1701993541,1701993543,1701993544,1701993546,1701993547,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701993541,1701993543,1701993544,1701993546,1701993547,1701993549,1701993550,0 2023-12-07T23:59:50.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first now 1701993541, trimmed 2023-12-07T23:59:50.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (23332 ms) 2023-12-07T23:59:50.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-12-07T23:59:50.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: foo0 2023-12-07T23:59:50.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-07T23:59:50.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (15796 ms) 2023-12-07T23:59:50.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-12-07T23:59:51.060 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:47.533 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.110 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:47.583 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.134 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:14:47.607 86737== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.169 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:47.614 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.174 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:47.616 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.181 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:47.653 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.184 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:47.657 86742== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.187 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:14:47.633 86583== Warning: unimplemented fcntl command: 1036 2023-12-07T23:59:51.202 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:14:47.674 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.195 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:02.667 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.199 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:02.672 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.234 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:02.707 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.245 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:02.690 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.246 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:02.691 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.249 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:02.722 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.250 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:02.723 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.274 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:02.719 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:06.279 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:02.752 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.139 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:17.611 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.141 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:17.614 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.168 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:17.641 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.187 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:17.660 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.188 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:17.633 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.189 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:17.634 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.191 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:17.665 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.212 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:17.657 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:21.217 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:17.690 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.290 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:18.763 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.291 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:18.764 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.304 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:18.777 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.355 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:18.800 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.358 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:18.801 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.370 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:18.814 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.437 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:18.910 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.438 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:18.911 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:22.456 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:18.929 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.483 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:27.955 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.488 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:27.961 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.513 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:27.986 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.542 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:27.987 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.545 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:28.017 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.545 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:27.989 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.545 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:28.018 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.572 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:28.017 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:31.573 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:28.046 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.379 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:37.852 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.383 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:37.856 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.418 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:37.890 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.447 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:37.891 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.461 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:37.933 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.462 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:37.935 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.502 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:37.975 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.522 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:37.968 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:41.586 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:38.031 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.480 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:47.952 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.482 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:47.956 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.502 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:47.975 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.531 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:47.976 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.532 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:47.978 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.553 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:48.025 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.555 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:47.998 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.555 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:48.027 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:00:51.578 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:48.051 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.361 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:57.834 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.363 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:57.837 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.385 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:15:57.858 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.411 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:57.884 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.413 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:57.858 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.414 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:57.887 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.415 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:57.860 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.432 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:15:57.905 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:01.435 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:15:57.881 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.394 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:07.867 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.396 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:07.870 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.412 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:07.886 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.440 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:07.885 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.442 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:07.888 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.455 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:07.928 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.460 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:07.905 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.460 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:07.934 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:11.475 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:07.948 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.598 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:18.071 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.601 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:18.075 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.602 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:18.075 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.626 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:18.100 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.669 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:18.142 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.698 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:18.170 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.725 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:18.170 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.735 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:18.181 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:21.761 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:18.206 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.481 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:27.954 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.485 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:27.959 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.514 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:27.987 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.529 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:27.974 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.537 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:27.978 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.541 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:28.014 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.544 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:28.017 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.553 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:27.999 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:31.563 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:28.036 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.565 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:33.038 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.617 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:33.090 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.632 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:33.105 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.664 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:33.109 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.666 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:33.111 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.670 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:33.143 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.672 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:33.145 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.678 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:33.123 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:36.683 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:33.156 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.563 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:48.036 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.569 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:48.042 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.598 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:48.071 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.610 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:48.056 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.613 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:48.059 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.618 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:48.091 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.620 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:48.093 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.641 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:48.087 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:51.650 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:48.121 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.553 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:48.998 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.554 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:48.999 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.562 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:16:49.008 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.580 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:49.053 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.582 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:49.054 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.590 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:16:49.064 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.619 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:49.092 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.620 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:49.093 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:01:52.630 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:49.103 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.572 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:08.045 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.621 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:08.094 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.624 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:08.069 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.627 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:08.073 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.646 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:08.119 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.649 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:08.095 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.672 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:08.145 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.675 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:08.148 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:11.695 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:08.168 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.354 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:08.799 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.355 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:08.800 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.362 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:08.808 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.398 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:08.871 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.399 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:08.872 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.417 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:08.890 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.419 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:08.892 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.420 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:08.893 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:12.428 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:08.901 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.695 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:28.168 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.697 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:28.171 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.721 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:28.194 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.725 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:28.198 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.741 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:28.187 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.743 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:28.188 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.749 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:28.222 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.770 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:28.214 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:31.785 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:28.258 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.652 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:29.098 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.653 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:29.099 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.662 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:29.108 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.764 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:29.237 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.765 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:29.238 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.774 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:29.247 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.813 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:29.286 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.814 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:29.288 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:32.825 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:29.298 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.629 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:43.102 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.637 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:43.111 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.643 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:43.117 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.649 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:43.122 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.652 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:43.097 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.655 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:43.101 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.764 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:43.238 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.772 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:43.245 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:46.784 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:43.229 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.637 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:44.110 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.638 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:44.112 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.648 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:44.121 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.704 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:44.149 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.704 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:44.150 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.713 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:44.159 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.951 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:44.424 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:47.958 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:44.428 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:48.054 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:44.527 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.881 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:48.355 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.886 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:48.359 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.899 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:48.373 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.934 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:48.407 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.936 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:48.382 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.937 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:48.410 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.938 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:48.384 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.950 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:17:48.423 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:02:51.953 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:17:48.399 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:06.921 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:03.394 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:06.926 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:03.399 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:06.950 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:03.424 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:06.976 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:03.422 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:06.980 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:03.426 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:07.001 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:03.446 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:07.001 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:03.474 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:07.006 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:03.479 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:07.023 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:03.497 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:09.653 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-08T00:03:09.638+0000 2267d640 -1 snap_mapper.add_oid found existing snaps mapped on 297:a968c35e:test-rados-api-smithi129-92769-80::foo:2, removing 2023-12-08T00:03:09.669 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-08T00:03:09.660+0000 2ae8e640 -1 snap_mapper.add_oid found existing snaps mapped on 297:a968c35e:test-rados-api-smithi129-92769-80::foo:2, removing 2023-12-08T00:03:12.096 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:08.569 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.099 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:08.572 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.107 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:08.580 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.112 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:08.557 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.122 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:08.595 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.155 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:08.600 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.168 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:08.613 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.179 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:08.652 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:12.180 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:08.654 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:21.957 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:18.430 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.006 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:18.479 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.036 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:18.509 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.057 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:18.530 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.058 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:18.503 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.059 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:18.505 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.062 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:18.535 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.084 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:18.530 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:22.091 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:18.564 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.005 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:28.478 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.010 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:28.484 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.040 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:28.513 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.056 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:28.501 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.058 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:28.532 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.059 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:28.504 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.063 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:28.536 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.085 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:28.530 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:32.088 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:28.561 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:36.981 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:33.454 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.027 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:33.473 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.028 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:33.474 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.029 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:33.503 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.030 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:33.503 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.031 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:33.504 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.046 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:33.491 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.050 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:33.523 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:37.050 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:33.524 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:47.982 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (22524 ms) 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (24816 ms) 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3776 ms) 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4740 ms) 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4461 ms) 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5507 ms) 2023-12-08T00:03:47.983 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5628 ms) 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40477 ms) 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18225 ms) 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5544 ms) 2023-12-08T00:03:47.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4573 ms) 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27772 ms) 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16196 ms) 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6403 ms) 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5436 ms) 2023-12-08T00:03:47.985 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrubs... 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26631 ms) 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12399 ms) 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11064 ms) 2023-12-08T00:03:47.986 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-12-08T00:03:47.987 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:03:47.987 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11343 ms) 2023-12-08T00:03:47.987 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-12-08T00:03:52.069 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:48.542 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.075 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:48.549 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.109 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:48.582 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.119 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:48.564 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.120 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:48.566 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.122 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:48.595 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.128 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:48.601 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.153 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:48.598 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:52.163 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:48.636 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.040 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:49.485 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.041 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:49.486 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.052 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:49.498 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.058 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:49.531 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.059 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:49.533 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.072 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:49.545 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.093 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:49.567 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.094 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:49.568 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:53.105 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:49.578 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.349 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:53.822 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.415 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:53.888 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.450 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:18:53.923 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.475 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:53.920 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.481 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:53.926 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.493 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:53.966 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.496 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:53.969 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.505 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:18:53.951 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:03:57.522 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:18:53.995 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.413 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:03.886 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.417 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:03.891 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.453 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:03.927 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.473 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:03.946 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.478 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:03.951 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.480 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:03.926 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.489 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:03.935 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.506 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:03.979 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:07.515 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:03.960 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.548 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:14.021 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.550 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:14.023 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.572 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:14.046 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.594 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:14.040 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.595 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:14.041 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.598 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:14.071 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.599 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:14.073 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.618 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:14.063 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:17.622 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:14.096 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.433 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:18.907 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.438 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:18.911 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.451 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:18.925 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.484 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:18.957 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.485 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:18.930 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.486 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:18.932 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.486 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:18.960 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.498 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:18.971 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:22.500 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:18.946 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.628 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:34.101 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.630 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:34.104 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.650 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:34.124 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.684 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:34.130 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.685 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:34.131 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.689 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:34.162 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.691 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:34.165 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.705 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:34.151 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:37.711 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:34.184 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.601 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:39.074 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.604 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:39.078 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.621 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:39.095 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.650 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:39.124 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.652 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:39.126 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.657 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:39.102 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.659 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:39.104 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.668 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:39.142 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:42.678 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:39.124 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:53.149 INFO:tasks.ceph.osd.7.smithi184.stderr:2023-12-08T00:04:53.116+0000 2227d640 -1 snap_mapper.add_oid found existing snaps mapped on 99:12a685a0:test-rados-api-smithi129-92769-97::foo:31, removing 2023-12-08T00:04:53.162 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-08T00:04:53.153+0000 24e82640 -1 snap_mapper.add_oid found existing snaps mapped on 99:12a685a0:test-rados-api-smithi129-92769-97::foo:31, removing 2023-12-08T00:04:57.546 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:54.019 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.606 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:54.079 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.629 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:54.103 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.648 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:54.094 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.653 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:54.098 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.665 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:54.138 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.666 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:54.139 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.672 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:54.117 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:57.688 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:54.161 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.318 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:54.764 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.321 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:54.766 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.328 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:54.774 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.359 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:54.832 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.360 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:54.833 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.375 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:54.848 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.379 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:54.852 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.380 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:54.853 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:04:58.387 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:54.861 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.117 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:59.590 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.124 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:59.598 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.129 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:59.575 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.132 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:59.578 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.133 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:59.607 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.136 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:59.609 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.198 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:19:59.644 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.202 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:19:59.676 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:03.206 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:19:59.679 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.637 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:09.110 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.641 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:09.115 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.669 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:09.142 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.690 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:09.164 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.691 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:09.137 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.693 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:09.138 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.694 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:09.167 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.716 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:09.161 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:12.718 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:09.192 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.713 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:14.186 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.761 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:14.207 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.762 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:14.208 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.770 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:14.243 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.782 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:14.227 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.796 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:14.270 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.827 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:14.300 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.830 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:14.303 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:17.848 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:14.321 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.714 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:24.187 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.720 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:24.193 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.745 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:24.219 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.765 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:24.211 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.767 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:24.213 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.774 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:24.248 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.779 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:24.253 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.792 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:24.237 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:27.805 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:24.278 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.738 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:34.212 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.741 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:34.215 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.768 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:34.241 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.787 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:34.232 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.789 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:34.234 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.791 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:34.264 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.793 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:34.267 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.813 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:34.259 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:37.818 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:34.291 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.874 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:39.348 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.880 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:39.353 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.905 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:39.378 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.926 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:39.372 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.929 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:39.375 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.936 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:39.409 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.937 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:39.410 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.949 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:39.394 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:42.954 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:39.427 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.894 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:54.367 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.897 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:54.371 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.934 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:54.407 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.946 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:54.392 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.947 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:54.393 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.948 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:54.422 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.950 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:54.424 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.980 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:54.426 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:57.996 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:54.469 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.773 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:55.216 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.773 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:55.217 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.780 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:20:55.226 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.843 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:55.316 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.843 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:55.317 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.853 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:20:55.326 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.912 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:55.386 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.913 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:55.387 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:05:58.997 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:20:55.470 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.031 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:09.504 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.077 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:09.523 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.078 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:09.524 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.084 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:09.558 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.111 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:09.557 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.118 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:09.592 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.134 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:09.607 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.135 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:09.609 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:13.168 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:09.641 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.029 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:14.502 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.030 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:14.504 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.048 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:14.521 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.074 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:14.519 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.078 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:14.551 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.079 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:14.524 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.081 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:14.554 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.093 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:14.538 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:18.096 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:14.569 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.044 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:24.517 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.048 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:24.522 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.079 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:24.553 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.100 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:24.545 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.104 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:24.577 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.105 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:24.550 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.111 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:24.584 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.141 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:24.587 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:28.151 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:24.624 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.194 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:34.668 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.197 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:34.671 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.238 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:34.711 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.250 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:34.695 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.250 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:34.696 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.260 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:34.733 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.265 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:34.738 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.279 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:34.724 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:38.297 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:34.770 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.086 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:39.559 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.087 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:39.561 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.098 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:39.571 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.133 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:39.606 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.133 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:39.579 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.134 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:39.580 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.136 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:39.609 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.147 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:39.593 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:43.152 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:39.626 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.127 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:54.600 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.129 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:54.603 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.151 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:54.624 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.174 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:54.620 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.175 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:54.621 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.177 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:54.651 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.180 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:54.653 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.199 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:54.645 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:58.200 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:54.673 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.162 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:55.608 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.163 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:55.609 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.176 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:55.621 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.200 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:55.674 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.201 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:55.675 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.212 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:55.685 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.246 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:55.720 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.247 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:55.721 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:06:59.263 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:55.737 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.030 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:57.503 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.034 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:57.508 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.050 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:57.524 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.051 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:57.525 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.056 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:57.502 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.103 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:21:57.576 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.111 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:57.556 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.120 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:21:57.593 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:01.158 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:21:57.604 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.389 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:09.862 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.393 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:09.867 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.423 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:09.897 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.441 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:09.887 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.442 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:09.915 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.443 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:09.889 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.444 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:09.917 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.469 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:09.915 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:13.472 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:09.945 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.371 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:14.844 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.423 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:14.868 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.427 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:14.873 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.440 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:14.914 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.446 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:14.892 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.464 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:14.938 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.494 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:14.967 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.498 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:14.971 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:18.518 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:14.991 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.379 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:24.853 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.384 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:24.857 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.384 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:24.857 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.422 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:24.896 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.451 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:24.924 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.456 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:24.902 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.462 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:24.908 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.489 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:24.963 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:28.493 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:24.939 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.524 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:34.969 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.524 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:34.998 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.525 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:34.970 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.528 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:35.002 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.560 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:35.006 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.569 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:35.042 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.597 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:35.071 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.602 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:35.076 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:38.642 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:35.115 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.515 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:39.988 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.518 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:39.991 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.523 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:39.968 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.524 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:39.969 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.538 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:39.984 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.539 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:40.013 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.572 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:40.045 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.575 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:40.048 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:43.591 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:40.064 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:47.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:07:47.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11325 ms) 2023-12-08T00:07:47.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7729 ms) 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6683 ms) 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27336 ms) 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15429 ms) 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-12-08T00:07:47.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (13355 ms) 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000010 69 73 |is| 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000012 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13762 ms) 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = reef 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7790 ms) 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (964884 ms total) 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (2116 ms) 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-12-08T00:07:47.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11550 ms) 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (21175 ms) 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (15 ms) 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (34856 ms total) 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (9115 ms) 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9124 ms) 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrub... 2023-12-08T00:07:47.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (25320 ms) 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11968 ms) 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8770 ms) 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8776 ms) 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11087 ms) 2023-12-08T00:07:47.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-12-08T00:07:58.602 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:55.076 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.606 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:55.080 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.634 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:22:55.108 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.656 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:55.129 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.659 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:55.132 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.660 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:55.106 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.661 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:55.107 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.691 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:22:55.136 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:07:58.699 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:22:55.172 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.573 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:10.047 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.575 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:10.049 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.607 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:10.081 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.622 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:10.068 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.624 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:10.069 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.625 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:10.098 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.628 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:10.102 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.654 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:10.100 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:13.659 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:10.133 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.610 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:20.083 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.611 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:20.085 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.639 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:20.113 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.658 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:20.104 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.660 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:20.106 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.661 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:20.134 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.664 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:20.137 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.684 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:20.129 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:23.688 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:20.161 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.735 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:25.208 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.800 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:25.273 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.823 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:25.297 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.847 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:25.292 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.848 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:25.321 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.849 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:25.295 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.853 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:25.326 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.869 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:25.315 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:28.879 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:25.351 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.712 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:40.186 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.720 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:40.194 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.765 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:40.211 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.767 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:40.241 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.769 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:40.214 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.781 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:40.254 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.785 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:40.258 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.797 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:40.243 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:43.829 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:40.302 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.570 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:41.016 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.571 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:41.017 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.580 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:41.025 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.689 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:41.162 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.690 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:41.163 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.700 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:41.173 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.713 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:41.186 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.714 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:41.188 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:44.728 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:41.201 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.353 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:50.827 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.357 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:50.803 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.359 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:50.833 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.360 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:50.806 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.368 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:50.841 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.373 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:50.847 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.394 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:50.840 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.403 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:50.876 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:54.411 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:50.884 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.809 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:55.283 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.812 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:55.286 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.832 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:23:55.305 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.871 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:55.314 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.872 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:55.318 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.884 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:55.357 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.886 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:55.360 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.888 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:23:55.334 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:08:58.902 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:23:55.375 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.831 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:00.304 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.832 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:00.306 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.845 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:00.319 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.880 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:00.353 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.881 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:00.327 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.881 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:00.354 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.882 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:00.328 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.894 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:00.367 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:03.895 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:00.340 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.003 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:10.476 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.005 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:10.479 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.021 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:10.494 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.049 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:10.522 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.052 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:10.526 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.069 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:10.514 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.070 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:10.516 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.073 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:10.546 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:14.090 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:10.536 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:23.981 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:20.454 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:23.989 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:20.463 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:24.012 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:20.486 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:24.032 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:20.506 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:24.033 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:20.507 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:24.037 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:20.483 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:24.042 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:20.487 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:24.057 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:20.530 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:24.066 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:20.511 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:28.973 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:25.446 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.034 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:25.507 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.053 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:25.527 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.084 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:25.557 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.084 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:25.530 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.086 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:25.559 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.086 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:25.532 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.103 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:25.549 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:29.111 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:25.584 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:43.971 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:40.444 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.021 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:40.494 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.049 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:40.522 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.064 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:40.510 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.066 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:40.540 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.070 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:40.516 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.071 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:40.545 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.098 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:40.571 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.098 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:40.543 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.773 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:41.219 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.774 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:41.220 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.782 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:41.228 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.799 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:41.272 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.800 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:41.273 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.805 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:41.278 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.805 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:41.279 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.809 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:41.282 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:44.814 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:41.288 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.179 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:55.652 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.188 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:55.661 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.228 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:55.674 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.230 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:55.704 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.233 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:55.679 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.248 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:55.722 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.251 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:55.725 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.264 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:24:55.709 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:09:59.279 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:24:55.752 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.124 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:00.598 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.125 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:00.599 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.139 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:00.613 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.172 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:00.645 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.176 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:00.650 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.182 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:00.628 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.184 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:00.630 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.197 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:00.670 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:04.200 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:00.645 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.172 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:10.645 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.176 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:10.650 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.204 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:10.678 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.228 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:10.673 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.228 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:10.701 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.230 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:10.676 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.236 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:10.709 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.246 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:10.692 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:14.250 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:10.723 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.198 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:20.668 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.199 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:20.673 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.252 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:20.726 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.257 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:20.730 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.261 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:20.735 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.265 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:20.711 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.284 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:20.728 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.286 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:20.760 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:24.309 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:20.754 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.443 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:25.917 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.445 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:25.919 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.461 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:25.935 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.490 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:25.935 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.492 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:25.938 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.502 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:25.975 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.503 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:25.977 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.508 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:25.954 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:29.523 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:25.996 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:39.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8978 ms) 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13198 ms) 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9856 ms) 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15019 ms) 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9859 ms) 2023-12-08T00:10:39.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9009 ms) 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: ok, hit_set contains 329:602f83fe:::foo:head 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9110 ms) 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,0 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first is 1701994141 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,0 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,0 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,0 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,0 2023-12-08T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,1701994146,1701994147,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,1701994146,1701994147,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,1701994146,1701994147,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,1701994146,1701994147,1701994149,1701994150,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,1701994146,1701994147,1701994149,1701994150,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994141,1701994143,1701994144,1701994146,1701994147,1701994149,1701994150,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1701994144,1701994146,1701994147,1701994149,1701994150,1701994153,1701994154,0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first now 1701994144, trimmed 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22261 ms) 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: foo0 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18257 ms) 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-12-08T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20650 ms) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23368 ms) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5443 ms) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4814 ms) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4503 ms) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4362 ms) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (262847 ms total) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1409215 ms total) 2023-12-08T00:10:39.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92423 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92423 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92595 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92595 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93109 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93109 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92817 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92817 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:10:39.363 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92682 2023-12-08T00:10:39.364 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92682 2023-12-08T00:10:39.364 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:10:39.364 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92496 2023-12-08T00:10:39.364 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92496 2023-12-08T00:10:39.364 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:10:39.364 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92750 2023-12-08T00:10:39.364 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92750 2023-12-08T00:10:44.550 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:41.023 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.552 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:41.025 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.612 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:41.085 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.612 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:41.057 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.615 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:41.061 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.638 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:41.108 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.638 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:41.109 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.650 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:41.096 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:44.674 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:41.147 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.378 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:41.823 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.378 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:41.824 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.385 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:41.859 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.386 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:25:41.832 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.386 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:41.860 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.395 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:25:41.869 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.436 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:41.909 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.436 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:41.910 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:10:45.444 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:25:41.918 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.687 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:01.160 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.693 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:01.167 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.738 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:01.184 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.740 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:01.214 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.745 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:01.219 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.745 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:01.219 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.747 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:01.193 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.782 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:01.228 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:04.788 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:01.262 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.558 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:02.004 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.559 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:02.005 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.567 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:02.013 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.712 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:02.186 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.713 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:02.187 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.722 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:02.195 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.738 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:02.212 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.739 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:02.213 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:05.748 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:02.221 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:08.931 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:05.405 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:08.934 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:05.408 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:08.948 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:05.422 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:08.981 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:05.427 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:08.983 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:05.429 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:08.988 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:05.461 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:08.991 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:05.465 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.000 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:05.446 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.005 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:05.479 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.647 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:06.121 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.655 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:06.128 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.658 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:06.104 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.663 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:06.137 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.703 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:06.149 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.707 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:06.181 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.715 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:06.188 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.718 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:06.164 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:09.728 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:06.201 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.731 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:21.204 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.732 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:21.206 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.746 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:21.220 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.774 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:21.219 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.776 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:21.222 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.787 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:21.260 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.790 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:21.236 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.790 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:21.263 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:24.804 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:21.278 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.330 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:21.803 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.331 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:21.805 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.332 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:21.778 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.334 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:21.779 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.339 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:21.813 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.340 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:21.786 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.365 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:21.838 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.366 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:21.840 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:25.374 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:21.847 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.789 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:41.262 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.790 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:41.263 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.805 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:41.279 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.834 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:41.279 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.838 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:41.307 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.839 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:41.283 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.839 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:41.310 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.852 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:41.298 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:44.853 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:41.327 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.392 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:41.838 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.393 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:41.839 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.400 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:26:41.846 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.407 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:41.881 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.408 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:41.882 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.416 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:26:41.890 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.441 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:41.914 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.442 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:41.915 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:11:45.452 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:26:41.925 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.834 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:01.307 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.838 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:01.312 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.855 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:01.328 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.885 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:01.330 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.888 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:01.361 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.889 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:01.363 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.890 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:01.336 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.905 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:01.378 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:04.908 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:01.354 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.455 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:01.900 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.456 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:01.902 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.460 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:01.933 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.461 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:01.935 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.464 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:01.910 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.477 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:01.950 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.477 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:01.951 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.478 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:01.951 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:05.486 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:01.959 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.870 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:21.343 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.875 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:21.347 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.892 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:21.366 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.921 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:21.367 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.922 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:21.368 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.926 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:21.399 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.931 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:21.405 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.947 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:21.420 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:24.950 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:21.396 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.519 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:21.965 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.520 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:21.994 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.520 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:21.966 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.522 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:21.995 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.528 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:21.974 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.542 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:22.016 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.544 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:22.017 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.546 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:22.020 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:25.551 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:22.025 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:44.928 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:41.401 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:44.932 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:41.406 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:44.953 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:41.427 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:44.982 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:41.427 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:44.984 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:41.430 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:44.998 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:41.471 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.001 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:41.447 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.015 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:41.488 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.032 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:41.506 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.582 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:42.028 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.583 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:42.029 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.585 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:42.058 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.585 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:42.059 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.591 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:27:42.037 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.596 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:27:42.070 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.618 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:42.091 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.620 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:42.092 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:12:45.630 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:27:42.103 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:04.978 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:01.451 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:04.979 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:01.453 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.003 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:01.477 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.029 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:01.474 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.030 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:01.476 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.031 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:01.504 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.035 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:01.509 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.047 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:01.493 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.052 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:01.525 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.627 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:02.072 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.627 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:02.073 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.635 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:02.081 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.662 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:02.136 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.663 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:02.137 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.671 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:02.145 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.689 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:02.162 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.690 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:02.163 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:05.697 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:02.171 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.043 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:21.516 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.045 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:21.519 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.064 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:21.538 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.092 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:21.565 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.092 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:21.538 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.093 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:21.567 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.095 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:21.541 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.110 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:21.583 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:25.111 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:21.556 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.056 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:26.530 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.058 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:26.532 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.068 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:26.542 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.100 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:26.546 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.104 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:26.550 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.106 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:26.580 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.108 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:26.581 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.115 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:26.561 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:30.117 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:26.590 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.089 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:31.563 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.091 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:31.565 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.102 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:31.576 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.135 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:31.608 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.137 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:31.583 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.139 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:31.612 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.140 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:31.586 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.152 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:31.625 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:35.153 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:31.599 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.106 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:41.580 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.111 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:41.585 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.129 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:41.602 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.154 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:41.600 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.159 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:41.605 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.160 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:41.634 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.164 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:41.637 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.171 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:41.616 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:45.178 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:41.652 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.100 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:46.573 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.147 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:46.593 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.148 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:46.621 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.150 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:46.596 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.168 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:28:46.614 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.168 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:46.642 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.197 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:46.670 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.201 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:46.674 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:13:50.213 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:28:46.686 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.133 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:01.607 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.180 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:01.654 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.196 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:01.670 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.225 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:01.671 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.228 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:01.674 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.233 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:01.706 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.234 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:01.708 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.242 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:01.688 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:05.248 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:01.721 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.149 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:06.622 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.195 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:06.669 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.213 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:06.686 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.244 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:06.690 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.245 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:06.691 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.257 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:06.702 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.259 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:06.733 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.261 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:06.735 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:10.272 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:06.744 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.227 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:16.701 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.229 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:16.702 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.246 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:16.720 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.277 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:16.722 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.281 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:16.727 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.294 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:16.739 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.296 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:16.769 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.300 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:16.773 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:20.313 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:16.786 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.235 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:21.708 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.236 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:21.710 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.249 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:21.722 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.288 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:21.733 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.290 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:21.735 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.300 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:21.746 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.300 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:21.774 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.304 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:21.777 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:25.315 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:21.788 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.440 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:22.913 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.444 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:22.918 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.450 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:22.895 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.452 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:22.898 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.458 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:22.931 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.460 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:22.934 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.466 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:22.940 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.468 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:22.913 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:26.480 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:22.953 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.290 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:26.761 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.335 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:26.809 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.347 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:26.820 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.380 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:26.825 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.382 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:26.828 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.393 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:26.838 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.397 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:26.871 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.399 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:26.873 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:30.410 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:26.883 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.319 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:41.793 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.324 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:41.798 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.339 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:41.813 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.371 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:41.816 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.374 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:41.819 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.388 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:41.834 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.393 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:41.866 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.394 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:41.867 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:45.409 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:41.882 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.400 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:46.874 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.401 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:46.875 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.412 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:29:46.885 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.446 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:46.892 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.447 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:46.893 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.448 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:46.921 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.449 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:46.922 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.456 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:29:46.902 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:14:50.459 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:29:46.932 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.403 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:01.877 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.407 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:01.881 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.422 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:01.896 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.453 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:01.926 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.456 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:01.929 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.456 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:01.902 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.459 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:01.905 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.471 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:01.944 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:05.472 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:01.918 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.412 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:06.885 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.414 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:06.888 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.427 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:06.900 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.460 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:06.905 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.461 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:06.935 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.462 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:06.908 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.463 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:06.937 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.473 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:06.919 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:10.477 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:06.950 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.457 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:21.931 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.460 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:21.934 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.477 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:21.950 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.513 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:21.958 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.513 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:21.959 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.524 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:21.998 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.527 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:22.000 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.527 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:21.973 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:25.541 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:22.015 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.468 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:26.942 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.469 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:26.943 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.478 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:26.952 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.510 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:26.983 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.512 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:26.958 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.514 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:26.960 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.514 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:26.988 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.524 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:26.970 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:30.525 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:26.999 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.459 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:41.932 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.508 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:41.954 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.509 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:41.982 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.511 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:41.957 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.528 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:42.002 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.528 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:41.974 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.561 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:42.035 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.563 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:42.036 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:45.577 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:42.050 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.511 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:46.984 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.516 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:46.990 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.528 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:30:47.002 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.559 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:47.005 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.561 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:47.007 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.562 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:47.036 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.565 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:47.039 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.571 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:30:47.017 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:15:50.576 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:30:47.049 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.557 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:02.030 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.558 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:02.032 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.572 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:02.046 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.606 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:02.052 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.607 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:02.080 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.608 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:02.053 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.610 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:02.084 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.622 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:02.068 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:05.625 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:02.099 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.651 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:12.124 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.656 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:12.129 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.674 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:12.147 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.702 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:12.148 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.703 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:12.149 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.706 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:12.179 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.707 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:12.181 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.718 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:12.163 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:15.722 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:12.195 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.664 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:17.138 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.666 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:17.140 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.678 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:17.152 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.710 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:17.156 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.713 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:17.159 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.723 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:17.169 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.731 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:17.204 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.732 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:17.206 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:20.743 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:17.217 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.705 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:32.178 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.709 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:32.182 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.725 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:32.198 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.759 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:32.232 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.762 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:32.235 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.762 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:32.207 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.764 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:32.210 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.776 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:32.249 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:35.779 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:32.225 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.714 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:37.187 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.717 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:37.190 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.729 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:37.203 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.760 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:37.205 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.762 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:37.208 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.764 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:37.238 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.766 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:37.239 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.773 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:37.218 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:40.776 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:37.250 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.751 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:52.225 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.755 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:52.229 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.771 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:52.245 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.806 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:52.252 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.807 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:52.253 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.809 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:52.282 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.810 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:52.284 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.821 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:52.267 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:16:55.824 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:52.298 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.761 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:57.234 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.768 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:57.242 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.781 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:31:57.255 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.810 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:57.256 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.813 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:57.259 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.816 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:57.289 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.817 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:57.291 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.824 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:31:57.269 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:00.828 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:31:57.302 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.798 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:12.272 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.802 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:12.276 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.821 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:12.294 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.849 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:12.294 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.850 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:12.296 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.851 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:12.325 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.854 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:12.328 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.865 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:12.311 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:15.871 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:12.345 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.814 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:17.287 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.816 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:17.290 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.828 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:17.302 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.858 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:17.304 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.859 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:17.332 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.862 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:17.335 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.862 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:17.308 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.874 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:17.347 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:20.874 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:17.320 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:21.974 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Running main() from gmock_main.cc 2023-12-08T00:17:21.974 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2023-12-08T00:17:21.974 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] Global test environment set-up. 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: handle_notify cookie 94118070231984 notify_id 4415226380289 notifier_gid 4588 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (1803381 ms) 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Trying... 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: handle_notify cookie 94118083040512 notify_id 4428111282176 notifier_gid 20176 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Waiting for 3.000000000s 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Timed out. 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Flushing... 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Flushed... 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (8112 ms) 2023-12-08T00:17:21.975 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (1811493 ms total) 2023-12-08T00:17:21.976 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: 2023-12-08T00:17:21.976 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] Global test environment tear-down 2023-12-08T00:17:21.976 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (1811493 ms total) 2023-12-08T00:17:21.976 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ PASSED ] 2 tests. 2023-12-08T00:17:34.343 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:30.816 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.349 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:30.822 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.350 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:30.796 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.352 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:30.798 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.372 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:30.817 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.378 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:30.851 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.388 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:30.861 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.390 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:30.863 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:34.411 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:30.884 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.824 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:32.297 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.831 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:32.305 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.835 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:32.308 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.877 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:32.323 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.878 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:32.324 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.886 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:32.331 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.903 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:32.376 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.923 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:32.396 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:35.936 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:32.409 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.894 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:37.368 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.896 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:37.370 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.911 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:37.385 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.943 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:37.388 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.944 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:37.390 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.945 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:37.419 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.947 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:37.420 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.956 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:37.402 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:40.961 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:37.435 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:48.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-12-08T00:17:48.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-12-08T00:17:48.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1816834 ms) 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (133 ms) 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1816967 ms total) 2023-12-08T00:17:48.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: 2023-12-08T00:17:48.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-12-08T00:17:48.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-12-08T00:17:48.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2023-12-08T00:17:48.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (446 ms) 2023-12-08T00:17:48.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-12-08T00:17:48.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2023-12-08T00:17:48.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3987 ms) 2023-12-08T00:17:48.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-12-08T00:17:48.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (223 ms) 2023-12-08T00:17:48.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-12-08T00:17:48.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (169 ms) 2023-12-08T00:17:48.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-12-08T00:17:48.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892432943248 notify_id 4475355922432 notifier_gid 15714 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (147 ms) 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433714336 notify_id 4475355922434 notifier_gid 15714 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (133 ms) 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433731664 notify_id 4475355922435 notifier_gid 15714 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (126 ms) 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-12-08T00:17:48.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433730400 notify_id 4475355922434 notifier_gid 15714 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (138 ms) 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433731664 notify_id 4475355922436 notifier_gid 15714 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (118 ms) 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433731664 notify_id 4475355922437 notifier_gid 15714 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (134 ms) 2023-12-08T00:17:48.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-12-08T00:17:48.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: trying... 2023-12-08T00:17:48.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433731664 notify_id 4475355922435 notifier_gid 15714 2023-12-08T00:17:48.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: timed out 2023-12-08T00:17:55.932 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:52.406 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:55.937 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:52.411 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:55.958 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:52.432 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:55.986 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:52.432 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:55.988 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:52.434 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.001 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:52.474 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.004 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:52.477 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.008 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:52.454 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.022 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:52.496 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.491 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:52.937 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.492 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:52.938 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.501 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:32:52.947 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.527 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:53.001 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.528 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:53.002 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.536 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:53.009 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.537 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:53.010 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.537 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:32:53.011 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:17:56.545 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:32:53.019 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:02.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2023-12-08T00:18:02.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushed 2023-12-08T00:18:02.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3086 ms) 2023-12-08T00:18:02.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-12-08T00:18:02.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: trying... 2023-12-08T00:18:02.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433731664 notify_id 4475355922436 notifier_gid 15714 2023-12-08T00:18:02.121 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: timed out 2023-12-08T00:18:02.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2023-12-08T00:18:02.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushed 2023-12-08T00:18:02.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3103 ms) 2023-12-08T00:18:02.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-12-08T00:18:02.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: List watches 2023-12-08T00:18:02.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433731664 notify_id 4475355922435 notifier_gid 15714 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 done 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: watch_check 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: unwatch2 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: done 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3121 ms) 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: List watches 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 2023-12-08T00:18:02.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 93892433729456 notify_id 4475355922438 notifier_gid 15714 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 done 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: watch_check 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: unwatch2 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: done 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3128 ms) 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18059 ms total) 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: 2023-12-08T00:18:02.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-12-08T00:18:02.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (1851922 ms total) 2023-12-08T00:18:02.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-12-08T00:18:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93272 2023-12-08T00:18:02.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93272 2023-12-08T00:18:02.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92514 2023-12-08T00:18:02.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92514 2023-12-08T00:18:02.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93143 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93143 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93021 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93021 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92397 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92397 2023-12-08T00:18:02.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92460 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92460 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93038 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93038 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92388 2023-12-08T00:18:02.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92388 2023-12-08T00:18:02.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92409 2023-12-08T00:18:02.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92409 2023-12-08T00:18:02.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93405 2023-12-08T00:18:02.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93405 2023-12-08T00:18:02.135 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92611 2023-12-08T00:18:02.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92611 2023-12-08T00:18:02.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93465 2023-12-08T00:18:02.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93465 2023-12-08T00:18:02.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92718 2023-12-08T00:18:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92718 2023-12-08T00:18:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93215 2023-12-08T00:18:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93215 2023-12-08T00:18:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93495 2023-12-08T00:18:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93495 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92648 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92648 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93082 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93082 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93186 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93186 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93446 2023-12-08T00:18:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93446 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92998 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92998 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92552 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92552 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93486 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93486 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92909 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92909 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92381 2023-12-08T00:18:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92381 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92442 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92442 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92801 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92801 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92479 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92479 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92922 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92922 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=92946 2023-12-08T00:18:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 92946 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93297 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93297 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=93529 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 93529 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ exit 0 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ cleanup 2023-12-08T00:18:02.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ pkill -P 92376 2023-12-08T00:18:02.142 INFO:tasks.workunit.client.0.smithi129.stderr:+ true 2023-12-08T00:18:02.143 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-08T00:18:02.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-08T00:18:02.174 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-12-08T00:18:02.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-08T00:18:02.516 DEBUG:teuthology.parallel:result is None 2023-12-08T00:18:02.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-08T00:18:02.544 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-08T00:18:02.544 DEBUG:teuthology.orchestra.run.smithi129:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-08T00:18:02.610 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-08T00:18:02.610 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-08T00:18:02.620 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-08T00:18:02.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-08T00:18:02.769 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.767+0000 7f45225f5640 10 monclient: get_monmap_and_config 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.767+0000 7f45225f5640 10 monclient: build_initial_monmap 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.767+0000 7f45225f5640 10 monclient: monmap: 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:02.768665+0000 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:02.768665+0000 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:02.770 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.768+0000 7f45225f5640 1 Processor -- start 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.768+0000 7f45225f5640 1 -- start start 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.768+0000 7f45225f5640 10 monclient: init 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f451c0a8950 0x7f451c0a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient(hunting): picked mon.noname-c con 0x7f451c0a8950 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f451c0b5610 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f451c13f110 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c141e30 con 0x7f451c0b5610 2023-12-08T00:18:02.772 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0aa890 con 0x7f451c13f110 2023-12-08T00:18:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f451c0b5a30 con 0x7f451c0a8950 2023-12-08T00:18:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f45225f5640 10 monclient(hunting): authenticate will time out at 3282.957209s 2023-12-08T00:18:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f4513fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f451c0b5610 0x7f451c1425a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:47978/0 (socket says 172.21.15.129:47978) 2023-12-08T00:18:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.769+0000 7f4513fff640 1 -- 172.21.15.129:0/2986593712 learned_addr learned my addr 172.21.15.129:0/2986593712 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.770+0000 7f451bfff640 1 --2- 172.21.15.129:0/2986593712 >> v2:172.21.15.184:3300/0 conn(0x7f451c0a8950 0x7f451c0a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:02.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.770+0000 7f451bfff640 10 monclient(hunting): get_auth_request con 0x7f451c0a8950 auth_method 0 2023-12-08T00:18:02.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.770+0000 7f451bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:02.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.770+0000 7f451bfff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:02.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.770+0000 7f451bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:02.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.771+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:02.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.771+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:02.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.771+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.772+0000 7f451b7fe640 1 -- 172.21.15.129:0/2986593712 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605915189 0 0) 0x7f451c141e30 con 0x7f451c0b5610 2023-12-08T00:18:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.772+0000 7f451b7fe640 10 monclient(hunting): _negotiate 2023-12-08T00:18:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.772+0000 7f451b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.772+0000 7f451b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.772+0000 7f451b7fe640 10 monclient(hunting): my global_id is 15717 2023-12-08T00:18:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.772+0000 7f451b7fe640 1 -- 172.21.15.129:0/2986593712 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4508003520 con 0x7f451c0b5610 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 10 monclient(hunting): handle_auth_done global_id 20185 payload 274 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 1 -- 172.21.15.129:0/2986593712 >> v1:172.21.15.184:6789/0 conn(0x7f451c13f110 legacy=0x7f451c14c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 1 -- 172.21.15.129:0/2986593712 >> v1:172.21.15.129:6789/0 conn(0x7f451c0b5610 legacy=0x7f451c1425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 1 monclient: found mon.noname-c 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.774+0000 7f451bfff640 1 -- 172.21.15.129:0/2986593712 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c142cb0 con 0x7f451c0a8950 2023-12-08T00:18:02.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.777+0000 7f451bfff640 1 --2- 172.21.15.129:0/2986593712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0a8950 0x7f451c0a8d30 secure :-1 s=READY pgs=4848 cs=0 l=1 rev1=1 crypto rx=0x7f450c0097a0 tx=0x7f450c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3caea6cae9ffe10 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f451b7fe640 1 -- 172.21.15.129:0/2986593712 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c030610 con 0x7f451c0a8950 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f451b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f451b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f451b7fe640 10 monclient: dump: 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:02.784 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:02.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f451b7fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:02.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f451b7fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:02.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f451b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:02.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f45225f5640 5 monclient: authenticate success, global_id 20185 2023-12-08T00:18:02.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.781+0000 7f45225f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f451b7fe640 1 -- 172.21.15.129:0/2986593712 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c030770 con 0x7f451c0a8950 2023-12-08T00:18:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f451b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f45225f5640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f451b7fe640 1 -- 172.21.15.129:0/2986593712 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c030ae0 con 0x7f451c0a8950 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f45225f5640 4 set_mon_vals no callback set 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f451b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f451b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f451b7fe640 10 monclient: dump: 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f45225f5640 10 monclient: shutdown 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f45225f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f45225f5640 1 -- 172.21.15.129:0/2986593712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0a8950 msgr2=0x7f451c0a8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.782+0000 7f45225f5640 1 --2- 172.21.15.129:0/2986593712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0a8950 0x7f451c0a8d30 secure :-1 s=READY pgs=4848 cs=0 l=1 rev1=1 crypto rx=0x7f450c0097a0 tx=0x7f450c02b960 comp rx=0 tx=0).stop 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.783+0000 7f45225f5640 1 -- 172.21.15.129:0/2986593712 shutdown_connections 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.783+0000 7f45225f5640 1 --2- 172.21.15.129:0/2986593712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0a8950 0x7f451c0a8d30 unknown :-1 s=CLOSED pgs=4848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.783+0000 7f45225f5640 1 -- 172.21.15.129:0/2986593712 >> 172.21.15.129:0/2986593712 conn(0x7f451c066cd0 msgr2=0x7f451c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.783+0000 7f45225f5640 1 -- 172.21.15.129:0/2986593712 shutdown_connections 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.783+0000 7f45225f5640 1 -- 172.21.15.129:0/2986593712 wait complete. 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.783+0000 7f45225f5640 10 monclient: build_initial_monmap 2023-12-08T00:18:02.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.783+0000 7f45225f5640 10 monclient: monmap: 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 librados: starting msgr at 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 librados: starting objecter 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 Processor -- start 2023-12-08T00:18:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 -- start start 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 librados: setting wanted keys 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 librados: calling monclient init 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient: init 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f451c0a8950 0x7f451c143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): picked mon.noname-c con 0x7f451c0a8950 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0b5610 0x7f451c1446e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): picked mon.noname-b con 0x7f451c0b5610 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f451c13f110 0x7f451c0b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): picked mon.noname-a con 0x7f451c13f110 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c143350 con 0x7f451c13f110 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c144e50 con 0x7f451c0b5610 2023-12-08T00:18:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451c144fe0 con 0x7f451c0a8950 2023-12-08T00:18:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.784+0000 7f45225f5640 10 monclient(hunting): authenticate will time out at 3282.972907s 2023-12-08T00:18:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.785+0000 7f451bfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f451c0a8950 0x7f451c143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.785+0000 7f451bfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f451c0a8950 0x7f451c143c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:58354/0 (socket says 172.21.15.129:58354) 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.785+0000 7f451bfff640 1 -- 172.21.15.129:0/3719834713 learned_addr learned my addr 172.21.15.129:0/3719834713 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.785+0000 7f451bfff640 10 monclient(hunting): get_auth_request con 0x7f451c0a8950 auth_method 0 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.785+0000 7f451bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.785+0000 7f451bfff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.785+0000 7f451bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f4513fff640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0b5610 0x7f451c1446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f4513fff640 10 monclient(hunting): get_auth_request con 0x7f451c0b5610 auth_method 0 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f4513fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f4513fff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.786+0000 7f4513fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f451c13f110 0x7f451c0b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 10 monclient(hunting): get_auth_request con 0x7f451c13f110 auth_method 0 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4513fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4513fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4513fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.787+0000 7f4520b92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 10 monclient(hunting): handle_auth_done global_id 18344 payload 824 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 1 -- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0b5610 msgr2=0x7f451c1446e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0b5610 0x7f451c1446e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 1 -- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f451c13f110 msgr2=0x7f451c0b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f451c13f110 0x7f451c0b07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 1 monclient: found mon.noname-c 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.789+0000 7f451bfff640 1 -- 172.21.15.129:0/3719834713 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c009400 con 0x7f451c0a8950 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.791+0000 7f451bfff640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f451c0a8950 0x7f451c143c20 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7f450c02b9a0 tx=0x7f450c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.791+0000 7f45197fa640 10 client.?.objecter ms_handle_connect 0x7f451c0a8950 2023-12-08T00:18:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.791+0000 7f45197fa640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:02.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 1 -- 172.21.15.129:0/3719834713 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c03f070 con 0x7f451c0a8950 2023-12-08T00:18:02.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:02.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:02.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: dump: 2023-12-08T00:18:02.797 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: _finish_auth 0 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: _check_auth_tickets 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 1 -- 172.21.15.129:0/3719834713 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c030e20 con 0x7f451c0a8950 2023-12-08T00:18:02.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 1 -- 172.21.15.129:0/3719834713 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c003830 con 0x7f451c0a8950 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 5 monclient: authenticate success, global_id 18344 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 10 monclient: _renew_subs 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c136550 con 0x7f451c0a8950 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f451affd640 4 set_mon_vals no callback set 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45197fa640 10 monclient: dump: 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 10 client.18344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 10 monclient: _renew_subs 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c0b0ec0 con 0x7f451c0a8950 2023-12-08T00:18:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.795+0000 7f45225f5640 1 librados: init done 2023-12-08T00:18:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.796+0000 7f45225f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.796+0000 7f45225f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.796+0000 7f45225f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.796+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451c0b72b0 con 0x7f451c0a8950 2023-12-08T00:18:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.813+0000 7f45197fa640 1 -- 172.21.15.129:0/3719834713 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f450c003990 con 0x7f451c0a8950 2023-12-08T00:18:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.813+0000 7f45197fa640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f44fc06c820 0x7f44fc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.814+0000 7f4513fff640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f44fc06c820 0x7f44fc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.814+0000 7f4513fff640 10 monclient: get_auth_request con 0x7f44fc06c820 auth_method 0 2023-12-08T00:18:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.814+0000 7f4513fff640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f44fc06c820 0x7f44fc06ece0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f451c145c60 tx=0x7f4504007d30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.814+0000 7f45197fa640 10 client.18344.objecter ms_handle_connect 0x7f44fc06c820 2023-12-08T00:18:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.816+0000 7f45197fa640 1 -- 172.21.15.129:0/3719834713 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(1046..1046 src has 978..1046) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f450c07a7e0 con 0x7f451c0a8950 2023-12-08T00:18:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.816+0000 7f45197fa640 10 client.18344.objecter ms_dispatch 0x7f451c000b90 osd_map(1046..1046 src has 978..1046) v4 2023-12-08T00:18:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.816+0000 7f45197fa640 3 client.18344.objecter handle_osd_map got epochs [1046,1046] > 0 2023-12-08T00:18:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.816+0000 7f45197fa640 3 client.18344.objecter handle_osd_map decoding full epoch 1046 2023-12-08T00:18:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:02.816+0000 7f45197fa640 20 client.18344.objecter dump_active .. 0 homeless 2023-12-08T00:18:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.051+0000 7f45197fa640 1 -- 172.21.15.129:0/3719834713 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f450c076b20 con 0x7f451c0a8950 2023-12-08T00:18:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.051+0000 7f45197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.051+0000 7f45197fa640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:03.226 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.223+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f451c142080 con 0x7f44fc06c820 2023-12-08T00:18:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.229+0000 7f45197fa640 1 -- 172.21.15.129:0/3719834713 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151287 (secure 0 0 0) 0x7f4504005850 con 0x7f44fc06c820 2023-12-08T00:18:03.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-08T00:18:03.234 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-08T00:18:03.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 10 librados: watch_flush enter 2023-12-08T00:18:03.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 10 librados: watch_flush exit 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 20 client.18344.objecter shutdown clearing up homeless session... 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 10 client.18344.objecter successfully canceled tick 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f44fc06c820 msgr2=0x7f44fc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f44fc06c820 0x7f44fc06ece0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f451c145c60 tx=0x7f4504007d30 comp rx=0 tx=0).stop 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 10 monclient: shutdown 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f451c0a8950 msgr2=0x7f451c143c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f451c0a8950 0x7f451c143c20 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7f450c02b9a0 tx=0x7f450c0046c0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 shutdown_connections 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f44fc06c820 0x7f44fc06ece0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f451c13f110 0x7f451c0b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f451c0b5610 0x7f451c1446e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 --2- 172.21.15.129:0/3719834713 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f451c0a8950 0x7f451c143c20 unknown :-1 s=CLOSED pgs=3479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 >> 172.21.15.129:0/3719834713 conn(0x7f451c066cd0 msgr2=0x7f451c0a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:03.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 shutdown_connections 2023-12-08T00:18:03.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 -- 172.21.15.129:0/3719834713 wait complete. 2023-12-08T00:18:03.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.231+0000 7f45225f5640 1 librados: shutdown 2023-12-08T00:18:03.246 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":2010,"stamp":"2023-12-08T00:18:02.135927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":97,"num_write":134,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":595240,"kb_used_data":54948,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754379480,"statfs":{"total":773094113280,"available":772484587520,"internally_reserved":0,"allocated":56266752,"data_stored":27347627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1792,"num_objects":-14,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14,"num_whiteouts":0,"num_read":-12,"num_read_kb":0,"num_write":-14,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000844"},"pg_stats":[{"pgid":"15.1f","version":"0'0","reported_seq":997,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.672927+0000","last_change":"2023-12-07T23:47:23.193178+0000","last_active":"2023-12-08T00:17:44.672927+0000","last_peered":"2023-12-08T00:17:44.672927+0000","last_clean":"2023-12-08T00:17:44.672927+0000","last_became_active":"2023-12-07T23:47:23.187382+0000","last_became_peered":"2023-12-07T23:47:23.187382+0000","last_unstale":"2023-12-08T00:17:44.672927+0000","last_undegraded":"2023-12-08T00:17:44.672927+0000","last_fullsized":"2023-12-08T00:17:44.672927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:46:32.619584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":990,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.069369+0000","last_change":"2023-12-07T23:47:26.349171+0000","last_active":"2023-12-08T00:17:45.069369+0000","last_peered":"2023-12-08T00:17:45.069369+0000","last_clean":"2023-12-08T00:17:45.069369+0000","last_became_active":"2023-12-07T23:47:26.088721+0000","last_became_peered":"2023-12-07T23:47:26.088721+0000","last_unstale":"2023-12-08T00:17:45.069369+0000","last_undegraded":"2023-12-08T00:17:45.069369+0000","last_fullsized":"2023-12-08T00:17:45.069369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:46:10.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1010,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.029374+0000","last_change":"2023-12-07T23:47:18.674241+0000","last_active":"2023-12-08T00:17:45.029374+0000","last_peered":"2023-12-08T00:17:45.029374+0000","last_clean":"2023-12-08T00:17:45.029374+0000","last_became_active":"2023-12-07T23:47:18.658074+0000","last_became_peered":"2023-12-07T23:47:18.658074+0000","last_unstale":"2023-12-08T00:17:45.029374+0000","last_undegraded":"2023-12-08T00:17:45.029374+0000","last_fullsized":"2023-12-08T00:17:45.029374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T00:31:54.317455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":987,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.798505+0000","last_change":"2023-12-07T23:47:39.142806+0000","last_active":"2023-12-08T00:18:00.798505+0000","last_peered":"2023-12-08T00:18:00.798505+0000","last_clean":"2023-12-08T00:18:00.798505+0000","last_became_active":"2023-12-07T23:47:39.133797+0000","last_became_peered":"2023-12-07T23:47:39.133797+0000","last_unstale":"2023-12-08T00:18:00.798505+0000","last_undegraded":"2023-12-08T00:18:00.798505+0000","last_fullsized":"2023-12-08T00:18:00.798505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:38:14.932982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":991,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.056223+0000","last_change":"2023-12-07T23:47:20.151369+0000","last_active":"2023-12-08T00:17:45.056223+0000","last_peered":"2023-12-08T00:17:45.056223+0000","last_clean":"2023-12-08T00:17:45.056223+0000","last_became_active":"2023-12-07T23:47:20.132690+0000","last_became_peered":"2023-12-07T23:47:20.132690+0000","last_unstale":"2023-12-08T00:17:45.056223+0000","last_undegraded":"2023-12-08T00:17:45.056223+0000","last_fullsized":"2023-12-08T00:17:45.056223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:51:11.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.029025+0000","last_change":"2023-12-07T23:47:19.053664+0000","last_active":"2023-12-08T00:17:45.029025+0000","last_peered":"2023-12-08T00:17:45.029025+0000","last_clean":"2023-12-08T00:17:45.029025+0000","last_became_active":"2023-12-07T23:47:18.926939+0000","last_became_peered":"2023-12-07T23:47:18.926939+0000","last_unstale":"2023-12-08T00:17:45.029025+0000","last_undegraded":"2023-12-08T00:17:45.029025+0000","last_fullsized":"2023-12-08T00:17:45.029025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:28:02.327836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1","version":"64'7","reported_seq":1006,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.725038+0000","last_change":"2023-12-07T23:47:15.440534+0000","last_active":"2023-12-08T00:18:00.725038+0000","last_peered":"2023-12-08T00:18:00.725038+0000","last_clean":"2023-12-08T00:18:00.725038+0000","last_became_active":"2023-12-07T23:47:15.386185+0000","last_became_peered":"2023-12-07T23:47:15.386185+0000","last_unstale":"2023-12-08T00:18:00.725038+0000","last_undegraded":"2023-12-08T00:18:00.725038+0000","last_fullsized":"2023-12-08T00:18:00.725038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:40:51.648331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":990,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.016890+0000","last_change":"2023-12-07T23:47:18.719911+0000","last_active":"2023-12-08T00:17:45.016890+0000","last_peered":"2023-12-08T00:17:45.016890+0000","last_clean":"2023-12-08T00:17:45.016890+0000","last_became_active":"2023-12-07T23:47:18.700591+0000","last_became_peered":"2023-12-07T23:47:18.700591+0000","last_unstale":"2023-12-08T00:17:45.016890+0000","last_undegraded":"2023-12-08T00:17:45.016890+0000","last_fullsized":"2023-12-08T00:17:45.016890+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:15:06.752653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.078513+0000","last_change":"2023-12-07T23:47:16.106412+0000","last_active":"2023-12-08T00:17:45.078513+0000","last_peered":"2023-12-08T00:17:45.078513+0000","last_clean":"2023-12-08T00:17:45.078513+0000","last_became_active":"2023-12-07T23:47:16.101044+0000","last_became_peered":"2023-12-07T23:47:16.101044+0000","last_unstale":"2023-12-08T00:17:45.078513+0000","last_undegraded":"2023-12-08T00:17:45.078513+0000","last_fullsized":"2023-12-08T00:17:45.078513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:51:45.972358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.067819+0000","last_change":"2023-12-07T23:47:19.725523+0000","last_active":"2023-12-08T00:17:45.067819+0000","last_peered":"2023-12-08T00:17:45.067819+0000","last_clean":"2023-12-08T00:17:45.067819+0000","last_became_active":"2023-12-07T23:47:18.123992+0000","last_became_peered":"2023-12-07T23:47:18.123992+0000","last_unstale":"2023-12-08T00:17:45.067819+0000","last_undegraded":"2023-12-08T00:17:45.067819+0000","last_fullsized":"2023-12-08T00:17:45.067819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:05:44.580089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":991,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.071367+0000","last_change":"2023-12-07T23:47:19.649219+0000","last_active":"2023-12-08T00:17:45.071367+0000","last_peered":"2023-12-08T00:17:45.071367+0000","last_clean":"2023-12-08T00:17:45.071367+0000","last_became_active":"2023-12-07T23:47:19.507339+0000","last_became_peered":"2023-12-07T23:47:19.507339+0000","last_unstale":"2023-12-08T00:17:45.071367+0000","last_undegraded":"2023-12-08T00:17:45.071367+0000","last_fullsized":"2023-12-08T00:17:45.071367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:38:46.882596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1007,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.639644+0000","last_change":"2023-12-07T23:48:06.632530+0000","last_active":"2023-12-08T00:17:44.639644+0000","last_peered":"2023-12-08T00:17:44.639644+0000","last_clean":"2023-12-08T00:17:44.639644+0000","last_became_active":"2023-12-07T23:48:06.627236+0000","last_became_peered":"2023-12-07T23:48:06.627236+0000","last_unstale":"2023-12-08T00:17:44.639644+0000","last_undegraded":"2023-12-08T00:17:44.639644+0000","last_fullsized":"2023-12-08T00:17:44.639644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:43:12.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1006,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.644941+0000","last_change":"2023-12-07T23:47:18.049591+0000","last_active":"2023-12-08T00:17:44.644941+0000","last_peered":"2023-12-08T00:17:44.644941+0000","last_clean":"2023-12-08T00:17:44.644941+0000","last_became_active":"2023-12-07T23:47:18.028936+0000","last_became_peered":"2023-12-07T23:47:18.028936+0000","last_unstale":"2023-12-08T00:17:44.644941+0000","last_undegraded":"2023-12-08T00:17:44.644941+0000","last_fullsized":"2023-12-08T00:17:44.644941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:11:43.305002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"841'88","reported_seq":1119,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.733737+0000","last_change":"2023-12-07T23:46:05.583767+0000","last_active":"2023-12-08T00:18:00.733737+0000","last_peered":"2023-12-08T00:18:00.733737+0000","last_clean":"2023-12-08T00:18:00.733737+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-08T00:18:00.733737+0000","last_undegraded":"2023-12-08T00:18:00.733737+0000","last_fullsized":"2023-12-08T00:18:00.733737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_clean_scrub_stamp":"2023-12-07T23:46:02.581722+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:48:03.670610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.057811+0000","last_change":"2023-12-07T23:46:13.815701+0000","last_active":"2023-12-08T00:17:45.057811+0000","last_peered":"2023-12-08T00:17:45.057811+0000","last_clean":"2023-12-08T00:17:45.057811+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-08T00:17:45.057811+0000","last_undegraded":"2023-12-08T00:17:45.057811+0000","last_fullsized":"2023-12-08T00:17:45.057811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:13.731531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":992,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.756380+0000","last_change":"2023-12-07T23:47:18.113515+0000","last_active":"2023-12-08T00:18:00.756380+0000","last_peered":"2023-12-08T00:18:00.756380+0000","last_clean":"2023-12-08T00:18:00.756380+0000","last_became_active":"2023-12-07T23:47:18.104193+0000","last_became_peered":"2023-12-07T23:47:18.104193+0000","last_unstale":"2023-12-08T00:18:00.756380+0000","last_undegraded":"2023-12-08T00:18:00.756380+0000","last_fullsized":"2023-12-08T00:18:00.756380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:39:47.005386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.013885+0000","last_change":"2023-12-07T23:46:13.418137+0000","last_active":"2023-12-08T00:17:45.013885+0000","last_peered":"2023-12-08T00:17:45.013885+0000","last_clean":"2023-12-08T00:17:45.013885+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-08T00:17:45.013885+0000","last_undegraded":"2023-12-08T00:17:45.013885+0000","last_fullsized":"2023-12-08T00:17:45.013885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:51:13.255892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1006,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.741729+0000","last_change":"2023-12-07T23:47:25.704244+0000","last_active":"2023-12-08T00:18:00.741729+0000","last_peered":"2023-12-08T00:18:00.741729+0000","last_clean":"2023-12-08T00:18:00.741729+0000","last_became_active":"2023-12-07T23:47:25.699009+0000","last_became_peered":"2023-12-07T23:47:25.699009+0000","last_unstale":"2023-12-08T00:18:00.741729+0000","last_undegraded":"2023-12-08T00:18:00.741729+0000","last_fullsized":"2023-12-08T00:18:00.741729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:23:03.360563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1001,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.639059+0000","last_change":"2023-12-07T23:46:13.343801+0000","last_active":"2023-12-08T00:17:44.639059+0000","last_peered":"2023-12-08T00:17:44.639059+0000","last_clean":"2023-12-08T00:17:44.639059+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-08T00:17:44.639059+0000","last_undegraded":"2023-12-08T00:17:44.639059+0000","last_fullsized":"2023-12-08T00:17:44.639059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:42:20.201085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1005,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.762618+0000","last_change":"2023-12-07T23:47:26.181304+0000","last_active":"2023-12-08T00:18:00.762618+0000","last_peered":"2023-12-08T00:18:00.762618+0000","last_clean":"2023-12-08T00:18:00.762618+0000","last_became_active":"2023-12-07T23:47:26.173821+0000","last_became_peered":"2023-12-07T23:47:26.173821+0000","last_unstale":"2023-12-08T00:18:00.762618+0000","last_undegraded":"2023-12-08T00:18:00.762618+0000","last_fullsized":"2023-12-08T00:18:00.762618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T23:47:56.552622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.028661+0000","last_change":"2023-12-07T23:46:13.873757+0000","last_active":"2023-12-08T00:17:45.028661+0000","last_peered":"2023-12-08T00:17:45.028661+0000","last_clean":"2023-12-08T00:17:45.028661+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-08T00:17:45.028661+0000","last_undegraded":"2023-12-08T00:17:45.028661+0000","last_fullsized":"2023-12-08T00:17:45.028661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:40:06.824492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1007,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.727055+0000","last_change":"2023-12-07T23:47:46.300836+0000","last_active":"2023-12-08T00:18:00.727055+0000","last_peered":"2023-12-08T00:18:00.727055+0000","last_clean":"2023-12-08T00:18:00.727055+0000","last_became_active":"2023-12-07T23:47:46.295177+0000","last_became_peered":"2023-12-07T23:47:46.295177+0000","last_unstale":"2023-12-08T00:18:00.727055+0000","last_undegraded":"2023-12-08T00:18:00.727055+0000","last_fullsized":"2023-12-08T00:18:00.727055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:54:00.508066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":994,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.055514+0000","last_change":"2023-12-07T23:46:13.905749+0000","last_active":"2023-12-08T00:17:45.055514+0000","last_peered":"2023-12-08T00:17:45.055514+0000","last_clean":"2023-12-08T00:17:45.055514+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-08T00:17:45.055514+0000","last_undegraded":"2023-12-08T00:17:45.055514+0000","last_fullsized":"2023-12-08T00:17:45.055514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:45:13.859049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":989,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.747705+0000","last_change":"2023-12-07T23:47:18.243486+0000","last_active":"2023-12-08T00:18:00.747705+0000","last_peered":"2023-12-08T00:18:00.747705+0000","last_clean":"2023-12-08T00:18:00.747705+0000","last_became_active":"2023-12-07T23:47:18.206551+0000","last_became_peered":"2023-12-07T23:47:18.206551+0000","last_unstale":"2023-12-08T00:18:00.747705+0000","last_undegraded":"2023-12-08T00:18:00.747705+0000","last_fullsized":"2023-12-08T00:18:00.747705+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:33:27.429730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.312242+0000","last_change":"2023-12-07T23:46:13.447293+0000","last_active":"2023-12-08T00:17:45.312242+0000","last_peered":"2023-12-08T00:17:45.312242+0000","last_clean":"2023-12-08T00:17:45.312242+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-08T00:17:45.312242+0000","last_undegraded":"2023-12-08T00:17:45.312242+0000","last_fullsized":"2023-12-08T00:17:45.312242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:00.005960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":988,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.039062+0000","last_change":"2023-12-07T23:47:15.786542+0000","last_active":"2023-12-08T00:17:45.039062+0000","last_peered":"2023-12-08T00:17:45.039062+0000","last_clean":"2023-12-08T00:17:45.039062+0000","last_became_active":"2023-12-07T23:47:15.770063+0000","last_became_peered":"2023-12-07T23:47:15.770063+0000","last_unstale":"2023-12-08T00:17:45.039062+0000","last_undegraded":"2023-12-08T00:17:45.039062+0000","last_fullsized":"2023-12-08T00:17:45.039062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:18:15.429980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1011,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.647483+0000","last_change":"2023-12-07T23:46:13.656681+0000","last_active":"2023-12-08T00:17:44.647483+0000","last_peered":"2023-12-08T00:17:44.647483+0000","last_clean":"2023-12-08T00:17:44.647483+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-08T00:17:44.647483+0000","last_undegraded":"2023-12-08T00:17:44.647483+0000","last_fullsized":"2023-12-08T00:17:44.647483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:29.359895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":989,"reported_epoch":1042,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.676520+0000","last_change":"2023-12-07T23:47:15.701780+0000","last_active":"2023-12-08T00:17:45.676520+0000","last_peered":"2023-12-08T00:17:45.676520+0000","last_clean":"2023-12-08T00:17:45.676520+0000","last_became_active":"2023-12-07T23:47:15.693158+0000","last_became_peered":"2023-12-07T23:47:15.693158+0000","last_unstale":"2023-12-08T00:17:45.676520+0000","last_undegraded":"2023-12-08T00:17:45.676520+0000","last_fullsized":"2023-12-08T00:17:45.676520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:12.790526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":995,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.078677+0000","last_change":"2023-12-07T23:46:13.620414+0000","last_active":"2023-12-08T00:17:45.078677+0000","last_peered":"2023-12-08T00:17:45.078677+0000","last_clean":"2023-12-08T00:17:45.078677+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-08T00:17:45.078677+0000","last_undegraded":"2023-12-08T00:17:45.078677+0000","last_fullsized":"2023-12-08T00:17:45.078677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:44:19.783064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":995,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.626794+0000","last_change":"2023-12-07T23:47:21.241228+0000","last_active":"2023-12-08T00:17:44.626794+0000","last_peered":"2023-12-08T00:17:44.626794+0000","last_clean":"2023-12-08T00:17:44.626794+0000","last_became_active":"2023-12-07T23:47:21.229799+0000","last_became_peered":"2023-12-07T23:47:21.229799+0000","last_unstale":"2023-12-08T00:17:44.626794+0000","last_undegraded":"2023-12-08T00:17:44.626794+0000","last_fullsized":"2023-12-08T00:17:44.626794+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:30:12.922388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":991,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.722464+0000","last_change":"2023-12-07T23:47:36.622692+0000","last_active":"2023-12-08T00:18:00.722464+0000","last_peered":"2023-12-08T00:18:00.722464+0000","last_clean":"2023-12-08T00:18:00.722464+0000","last_became_active":"2023-12-07T23:47:36.616695+0000","last_became_peered":"2023-12-07T23:47:36.616695+0000","last_unstale":"2023-12-08T00:18:00.722464+0000","last_undegraded":"2023-12-08T00:18:00.722464+0000","last_fullsized":"2023-12-08T00:18:00.722464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:48:24.932348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":991,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.731548+0000","last_change":"2023-12-07T23:47:36.574414+0000","last_active":"2023-12-08T00:18:00.731548+0000","last_peered":"2023-12-08T00:18:00.731548+0000","last_clean":"2023-12-08T00:18:00.731548+0000","last_became_active":"2023-12-07T23:47:36.567130+0000","last_became_peered":"2023-12-07T23:47:36.567130+0000","last_unstale":"2023-12-08T00:18:00.731548+0000","last_undegraded":"2023-12-08T00:18:00.731548+0000","last_fullsized":"2023-12-08T00:18:00.731548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:53:21.050211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.054680+0000","last_change":"2023-12-07T23:47:37.286320+0000","last_active":"2023-12-08T00:17:45.054680+0000","last_peered":"2023-12-08T00:17:45.054680+0000","last_clean":"2023-12-08T00:17:45.054680+0000","last_became_active":"2023-12-07T23:47:37.279080+0000","last_became_peered":"2023-12-07T23:47:37.279080+0000","last_unstale":"2023-12-08T00:17:45.054680+0000","last_undegraded":"2023-12-08T00:17:45.054680+0000","last_fullsized":"2023-12-08T00:17:45.054680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:43:22.378569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":990,"reported_epoch":1042,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.866514+0000","last_change":"2023-12-07T23:47:21.394853+0000","last_active":"2023-12-08T00:17:45.866514+0000","last_peered":"2023-12-08T00:17:45.866514+0000","last_clean":"2023-12-08T00:17:45.866514+0000","last_became_active":"2023-12-07T23:47:21.389382+0000","last_became_peered":"2023-12-07T23:47:21.389382+0000","last_unstale":"2023-12-08T00:17:45.866514+0000","last_undegraded":"2023-12-08T00:17:45.866514+0000","last_fullsized":"2023-12-08T00:17:45.866514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:18.044930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1005,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.751316+0000","last_change":"2023-12-07T23:47:20.975428+0000","last_active":"2023-12-08T00:18:00.751316+0000","last_peered":"2023-12-08T00:18:00.751316+0000","last_clean":"2023-12-08T00:18:00.751316+0000","last_became_active":"2023-12-07T23:47:20.968485+0000","last_became_peered":"2023-12-07T23:47:20.968485+0000","last_unstale":"2023-12-08T00:18:00.751316+0000","last_undegraded":"2023-12-08T00:18:00.751316+0000","last_fullsized":"2023-12-08T00:18:00.751316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:03:02.615974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":987,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.786637+0000","last_change":"2023-12-07T23:47:21.404597+0000","last_active":"2023-12-08T00:18:00.786637+0000","last_peered":"2023-12-08T00:18:00.786637+0000","last_clean":"2023-12-08T00:18:00.786637+0000","last_became_active":"2023-12-07T23:47:21.395304+0000","last_became_peered":"2023-12-07T23:47:21.395304+0000","last_unstale":"2023-12-08T00:18:00.786637+0000","last_undegraded":"2023-12-08T00:18:00.786637+0000","last_fullsized":"2023-12-08T00:18:00.786637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:02:02.413893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1004,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.759930+0000","last_change":"2023-12-07T23:47:38.795823+0000","last_active":"2023-12-08T00:18:00.759930+0000","last_peered":"2023-12-08T00:18:00.759930+0000","last_clean":"2023-12-08T00:18:00.759930+0000","last_became_active":"2023-12-07T23:47:38.790323+0000","last_became_peered":"2023-12-07T23:47:38.790323+0000","last_unstale":"2023-12-08T00:18:00.759930+0000","last_undegraded":"2023-12-08T00:18:00.759930+0000","last_fullsized":"2023-12-08T00:18:00.759930+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:40:01.316349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":990,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.015499+0000","last_change":"2023-12-07T23:47:37.187628+0000","last_active":"2023-12-08T00:17:45.015499+0000","last_peered":"2023-12-08T00:17:45.015499+0000","last_clean":"2023-12-08T00:17:45.015499+0000","last_became_active":"2023-12-07T23:47:37.180489+0000","last_became_peered":"2023-12-07T23:47:37.180489+0000","last_unstale":"2023-12-08T00:17:45.015499+0000","last_undegraded":"2023-12-08T00:17:45.015499+0000","last_fullsized":"2023-12-08T00:17:45.015499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:28:21.180560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1006,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.617573+0000","last_change":"2023-12-07T23:48:09.211978+0000","last_active":"2023-12-08T00:17:44.617573+0000","last_peered":"2023-12-08T00:17:44.617573+0000","last_clean":"2023-12-08T00:17:44.617573+0000","last_became_active":"2023-12-07T23:48:09.186728+0000","last_became_peered":"2023-12-07T23:48:09.186728+0000","last_unstale":"2023-12-08T00:17:44.617573+0000","last_undegraded":"2023-12-08T00:17:44.617573+0000","last_fullsized":"2023-12-08T00:17:44.617573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:09:04.978015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":988,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.764294+0000","last_change":"2023-12-07T23:47:20.131095+0000","last_active":"2023-12-08T00:18:00.764294+0000","last_peered":"2023-12-08T00:18:00.764294+0000","last_clean":"2023-12-08T00:18:00.764294+0000","last_became_active":"2023-12-07T23:47:20.123312+0000","last_became_peered":"2023-12-07T23:47:20.123312+0000","last_unstale":"2023-12-08T00:18:00.764294+0000","last_undegraded":"2023-12-08T00:18:00.764294+0000","last_fullsized":"2023-12-08T00:18:00.764294+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:27:53.246959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1006,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.636698+0000","last_change":"2023-12-07T23:47:18.960597+0000","last_active":"2023-12-08T00:17:44.636698+0000","last_peered":"2023-12-08T00:17:44.636698+0000","last_clean":"2023-12-08T00:17:44.636698+0000","last_became_active":"2023-12-07T23:47:18.946564+0000","last_became_peered":"2023-12-07T23:47:18.946564+0000","last_unstale":"2023-12-08T00:17:44.636698+0000","last_undegraded":"2023-12-08T00:17:44.636698+0000","last_fullsized":"2023-12-08T00:17:44.636698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:55:18.226923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":2,"acting":2,"num_store_stats":2},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477119,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82556,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289284,"statfs":{"total":96636764160,"available":96552226816,"internally_reserved":0,"allocated":7131136,"data_stored":3426749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":8.3040000000000003,"5min":8.2520000000000007,"15min":15.449999999999999},"min":{"1min":4.4939999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"max":{"1min":20.571999999999999,"5min":20.571999999999999,"15min":272.77100000000002},"last":17.207999999999998},{"interface":"front","average":{"1min":11.718,"5min":10.836,"15min":18.363},"min":{"1min":5.5990000000000002,"5min":4.9900000000000002,"15min":4.9649999999999999},"max":{"1min":47.829999999999998,"5min":47.829999999999998,"15min":288.42000000000002},"last":17.963000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.33,"5min":11.922000000000001,"15min":21.003},"min":{"1min":6.2279999999999998,"5min":6.2279999999999998,"15min":5.7649999999999997},"max":{"1min":73.218999999999994,"5min":73.218999999999994,"15min":320.57900000000001},"last":10.895},{"interface":"front","average":{"1min":16.672000000000001,"5min":13.327,"15min":23.423999999999999},"min":{"1min":9.1400000000000006,"5min":6.7279999999999998,"15min":5.5670000000000002},"max":{"1min":91.882000000000005,"5min":91.882000000000005,"15min":461.51799999999997},"last":20.463000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:38 2023","interfaces":[{"interface":"back","average":{"1min":11.853,"5min":9.7810000000000006,"15min":15.744},"min":{"1min":7.343,"5min":6.0389999999999997,"15min":5.6959999999999997},"max":{"1min":66.941999999999993,"5min":66.941999999999993,"15min":246.47499999999999},"last":19.763999999999999},{"interface":"front","average":{"1min":17.085000000000001,"5min":14.343,"15min":22.997},"min":{"1min":8.7669999999999995,"5min":8.7279999999999998,"15min":7.3079999999999998},"max":{"1min":69.078999999999994,"5min":69.078999999999994,"15min":356.02100000000002},"last":21.995999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":17.209,"5min":14.24,"15min":19.495999999999999},"min":{"1min":7.7300000000000004,"5min":7.048,"15min":6.7690000000000001},"max":{"1min":107.04900000000001,"5min":107.04900000000001,"15min":180.76400000000001},"last":14.782999999999999},{"interface":"front","average":{"1min":16.361000000000001,"5min":14.583,"15min":22.693000000000001},"min":{"1min":10.125,"5min":6.5629999999999997,"15min":6.2990000000000004},"max":{"1min":62.020000000000003,"5min":62.020000000000003,"15min":223.91300000000001},"last":21.356999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.823,"5min":15.294,"15min":24.542000000000002},"min":{"1min":10.445,"5min":8.9920000000000009,"15min":7.2889999999999997},"max":{"1min":19.890000000000001,"5min":29.309999999999999,"15min":251.892},"last":18.597999999999999},{"interface":"front","average":{"1min":29.396999999999998,"5min":19.707000000000001,"15min":28.649999999999999},"min":{"1min":13.223000000000001,"5min":11.711,"15min":10.568},"max":{"1min":265.69799999999998,"5min":265.69799999999998,"15min":279.791},"last":23.388000000000002}]},{"osd":6,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":21.244,"5min":14.326000000000001,"15min":19.547000000000001},"min":{"1min":7.2850000000000001,"5min":7.2850000000000001,"15min":7.2850000000000001},"max":{"1min":214.774,"5min":214.774,"15min":214.774},"last":12.128},{"interface":"front","average":{"1min":26.649999999999999,"5min":17.835000000000001,"15min":24.134},"min":{"1min":12.226000000000001,"5min":10.428000000000001,"15min":8.1440000000000001},"max":{"1min":229.54300000000001,"5min":229.54300000000001,"15min":229.54300000000001},"last":13.499000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":16.917000000000002,"5min":17.433,"15min":28.428000000000001},"min":{"1min":8.923,"5min":8.923,"15min":8.923},"max":{"1min":21.625,"5min":27.763999999999999,"15min":281.31},"last":24.106000000000002},{"interface":"front","average":{"1min":14.558999999999999,"5min":14.391999999999999,"15min":23.718},"min":{"1min":9.9049999999999994,"5min":8.5600000000000005,"15min":7.649},"max":{"1min":23.234999999999999,"5min":23.234999999999999,"15min":255.72499999999999},"last":15.93}]}]},{"osd":2,"up_from":16,"seq":68719477120,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69880,"kb_used_data":6292,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301960,"statfs":{"total":96636764160,"available":96565207040,"internally_reserved":0,"allocated":6443008,"data_stored":3053431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5220000000000002,"5min":8.8870000000000005,"15min":15.034000000000001},"min":{"1min":4.8140000000000001,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":20.408000000000001,"5min":20.408000000000001,"15min":249.42699999999999},"last":15.491},{"interface":"front","average":{"1min":9.9410000000000007,"5min":9.4969999999999999,"15min":15.816000000000001},"min":{"1min":5.9480000000000004,"5min":5.8230000000000004,"15min":5.1310000000000002},"max":{"1min":19.068999999999999,"5min":19.068999999999999,"15min":241.45599999999999},"last":11.285}]},{"osd":1,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":12.634,"5min":11.613,"15min":19.722000000000001},"min":{"1min":8.1159999999999997,"5min":6.8179999999999996,"15min":6.766},"max":{"1min":19.122,"5min":19.122,"15min":352.21100000000001},"last":12.108000000000001},{"interface":"front","average":{"1min":11.063000000000001,"5min":10.92,"15min":17.779},"min":{"1min":8.3200000000000003,"5min":6.2720000000000002,"15min":5.9109999999999996},"max":{"1min":14.936,"5min":18.652000000000001,"15min":327.81099999999998},"last":13.92}]},{"osd":3,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.925000000000001,"5min":10.75,"15min":19.754000000000001},"min":{"1min":7.798,"5min":6.2770000000000001,"15min":6.1550000000000002},"max":{"1min":15.944000000000001,"5min":19.123999999999999,"15min":287.56299999999999},"last":8.1010000000000009},{"interface":"front","average":{"1min":14.613,"5min":13.928000000000001,"15min":21.443999999999999},"min":{"1min":8.9900000000000002,"5min":8.3059999999999992,"15min":7.149},"max":{"1min":17.920000000000002,"5min":20.739000000000001,"15min":255.334},"last":16.969000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":15.124000000000001,"5min":15.16,"15min":23.259},"min":{"1min":8.9789999999999992,"5min":8.9220000000000006,"15min":8.1539999999999999},"max":{"1min":19.739999999999998,"5min":20.571000000000002,"15min":301.12799999999999},"last":20.748999999999999},{"interface":"front","average":{"1min":12.436999999999999,"5min":12.302,"15min":20.571000000000002},"min":{"1min":6.7450000000000001,"5min":6.7450000000000001,"15min":6.2649999999999997},"max":{"1min":19.678999999999998,"5min":23.859999999999999,"15min":348.38099999999997},"last":10.226000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":16.494,"15min":24.294},"min":{"1min":14.236000000000001,"5min":9.0410000000000004,"15min":8.5470000000000006},"max":{"1min":20.163,"5min":22.094000000000001,"15min":289.18700000000001},"last":23.734000000000002},{"interface":"front","average":{"1min":14.500999999999999,"5min":14.137,"15min":20.013000000000002},"min":{"1min":9.3870000000000005,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":20.331,"5min":20.667000000000002,"15min":248.63499999999999},"last":21.385000000000002}]},{"osd":6,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":16.123000000000001,"5min":15.634,"15min":21.067},"min":{"1min":10.611000000000001,"5min":9.9610000000000003,"15min":7.4240000000000004},"max":{"1min":19.946000000000002,"5min":21.585000000000001,"15min":349.41399999999999},"last":41.552999999999997},{"interface":"front","average":{"1min":11.491,"5min":12.48,"15min":20.643999999999998},"min":{"1min":7.431,"5min":7.423,"15min":7.423},"max":{"1min":19.212,"5min":21.007000000000001,"15min":231.422},"last":34.932000000000002}]},{"osd":7,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":13.156000000000001,"5min":14.472,"15min":24.436},"min":{"1min":8.6010000000000009,"5min":8.6010000000000009,"15min":8.6010000000000009},"max":{"1min":20.013999999999999,"5min":20.428000000000001,"15min":350.52199999999999},"last":16.106999999999999},{"interface":"front","average":{"1min":16.969000000000001,"5min":17.178000000000001,"15min":24.536000000000001},"min":{"1min":11.411,"5min":10.608000000000001,"15min":10.475},"max":{"1min":20.597000000000001,"5min":22.510999999999999,"15min":235.19200000000001},"last":14.58}]}]},{"osd":0,"up_from":16,"seq":68719477119,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70076,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301764,"statfs":{"total":96636764160,"available":96565006336,"internally_reserved":0,"allocated":7131136,"data_stored":3426749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 8 00:17:29 2023","interfaces":[{"interface":"back","average":{"1min":10.356,"5min":9.7119999999999997,"15min":18.596},"min":{"1min":5.2789999999999999,"5min":4.827,"15min":4.5830000000000002},"max":{"1min":38.834000000000003,"5min":38.834000000000003,"15min":317.99599999999998},"last":14.888999999999999},{"interface":"front","average":{"1min":11.698,"5min":9.2210000000000001,"15min":17.318999999999999},"min":{"1min":6.6840000000000002,"5min":4.9139999999999997,"15min":4.9139999999999997},"max":{"1min":49.262,"5min":49.262,"15min":313.14400000000001},"last":7.2000000000000002}]},{"osd":2,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":30.468,"5min":16.053000000000001,"15min":20.640000000000001},"min":{"1min":8.9280000000000008,"5min":7.1600000000000001,"15min":6.4740000000000002},"max":{"1min":196.96700000000001,"5min":196.96700000000001,"15min":228.02500000000001},"last":11.413},{"interface":"front","average":{"1min":23.956,"5min":14.147,"15min":18.375},"min":{"1min":6.8819999999999997,"5min":5.7530000000000001,"15min":5.6059999999999999},"max":{"1min":168.91300000000001,"5min":168.91300000000001,"15min":197.78899999999999},"last":8.3989999999999991}]},{"osd":3,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":14.613,"5min":13.257,"15min":22.777999999999999},"min":{"1min":10.614000000000001,"5min":8.0500000000000007,"15min":7.0069999999999997},"max":{"1min":19.207999999999998,"5min":19.207999999999998,"15min":239.50999999999999},"last":12.526},{"interface":"front","average":{"1min":16.495999999999999,"5min":16.079000000000001,"15min":24.495999999999999},"min":{"1min":10.835000000000001,"5min":10.667,"15min":9.141},"max":{"1min":20.774000000000001,"5min":29.132999999999999,"15min":241.839},"last":15.701000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":8.8249999999999993,"5min":8.9369999999999994,"15min":14.958},"min":{"1min":6.4029999999999996,"5min":5.5570000000000004,"15min":5.5430000000000001},"max":{"1min":22.925000000000001,"5min":22.925000000000001,"15min":211.452},"last":10.794},{"interface":"front","average":{"1min":16.763999999999999,"5min":16.686,"15min":22.658000000000001},"min":{"1min":10.212999999999999,"5min":10.212999999999999,"15min":9.3030000000000008},"max":{"1min":24.173999999999999,"5min":29.969000000000001,"15min":212.459},"last":16.332000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":21.459,"5min":14.852,"15min":19.033999999999999},"min":{"1min":8.6750000000000007,"5min":7.8970000000000002,"15min":5.9740000000000002},"max":{"1min":175.68100000000001,"5min":175.68100000000001,"15min":175.68100000000001},"last":9.8079999999999998},{"interface":"front","average":{"1min":22.960000000000001,"5min":13.959,"15min":19.902000000000001},"min":{"1min":8.1950000000000003,"5min":6.7789999999999999,"15min":6.6660000000000004},"max":{"1min":186.625,"5min":186.625,"15min":186.625},"last":13.194000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":17.789999999999999,"15min":26.085000000000001},"min":{"1min":11.406000000000001,"5min":11.406000000000001,"15min":10.641999999999999},"max":{"1min":26.757999999999999,"5min":30.727,"15min":227.892},"last":17.099},{"interface":"front","average":{"1min":16.129999999999999,"5min":14.592000000000001,"15min":20.123999999999999},"min":{"1min":10.122,"5min":8.2919999999999998,"15min":8.2919999999999998},"max":{"1min":44.432000000000002,"5min":44.432000000000002,"15min":153.59700000000001},"last":14.284000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.475000000000001,"5min":16.189,"15min":24.940000000000001},"min":{"1min":11.638,"5min":9.3550000000000004,"15min":8.9369999999999994},"max":{"1min":39.835999999999999,"5min":39.835999999999999,"15min":264.44600000000003},"last":19.535},{"interface":"front","average":{"1min":14.044,"5min":12.993,"15min":20.210000000000001},"min":{"1min":7.8209999999999997,"5min":7.5350000000000001,"15min":7.5},"max":{"1min":41.091000000000001,"5min":41.091000000000001,"15min":243.84899999999999},"last":17.846}]}]},{"osd":1,"up_from":14,"seq":60129542525,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70520,"kb_used_data":6936,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301320,"statfs":{"total":96636764160,"available":96564551680,"internally_reserved":0,"allocated":7102464,"data_stored":3577342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.6829999999999998,"5min":9.3740000000000006,"15min":17.443000000000001},"min":{"1min":5.8540000000000001,"5min":5.6449999999999996,"15min":4.2539999999999996},"max":{"1min":17.231000000000002,"5min":18.986999999999998,"15min":324.46600000000001},"last":7.7610000000000001},{"interface":"front","average":{"1min":14.244,"5min":13.406000000000001,"15min":22.675000000000001},"min":{"1min":7.2469999999999999,"5min":6.4950000000000001,"15min":5.5369999999999999},"max":{"1min":21.832000000000001,"5min":21.832000000000001,"15min":342.39299999999997},"last":10.605}]},{"osd":2,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.9380000000000006,"5min":9.7550000000000008,"15min":20.015000000000001},"min":{"1min":5.6289999999999996,"5min":4.7409999999999997,"15min":4.7409999999999997},"max":{"1min":15.329000000000001,"5min":18.332000000000001,"15min":309.75999999999999},"last":83.418999999999997},{"interface":"front","average":{"1min":13.932,"5min":13.237,"15min":23.968},"min":{"1min":7.7919999999999998,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":19.77,"5min":22.640999999999998,"15min":343.49900000000002},"last":17.074999999999999}]},{"osd":3,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":13.263,"5min":14.33,"15min":22.765000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"max":{"1min":17.984999999999999,"5min":20.395,"15min":345.07600000000002},"last":11.33},{"interface":"front","average":{"1min":10.297000000000001,"5min":9.077,"15min":15.353},"min":{"1min":7.5209999999999999,"5min":6.1050000000000004,"15min":5.3090000000000002},"max":{"1min":15.089,"5min":16.981999999999999,"15min":174.20599999999999},"last":8.0690000000000008}]},{"osd":4,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.705,"5min":11.641999999999999,"15min":19.434999999999999},"min":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.0419999999999998},"max":{"1min":18.649999999999999,"5min":19.026,"15min":290.358},"last":12.401999999999999},{"interface":"front","average":{"1min":12.657999999999999,"5min":12.154999999999999,"15min":19.594999999999999},"min":{"1min":7.8029999999999999,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":17.945,"5min":19.114000000000001,"15min":203.929},"last":15.039999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":32.521999999999998,"5min":19.515000000000001,"15min":24.640999999999998},"min":{"1min":10.952999999999999,"5min":10.938000000000001,"15min":7.7690000000000001},"max":{"1min":333.11799999999999,"5min":333.11799999999999,"15min":371.50599999999997},"last":13.135999999999999},{"interface":"front","average":{"1min":28.724,"5min":15.845000000000001,"15min":19.704000000000001},"min":{"1min":9.0389999999999997,"5min":7.6150000000000002,"15min":7.282},"max":{"1min":298.12099999999998,"5min":298.12099999999998,"15min":298.12099999999998},"last":8.6370000000000005}]},{"osd":6,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":14.052,"5min":13.733000000000001,"15min":19.829000000000001},"min":{"1min":9.0879999999999992,"5min":7.3490000000000002,"15min":7.3490000000000002},"max":{"1min":19.085000000000001,"5min":24.93,"15min":292.72300000000001},"last":16.273},{"interface":"front","average":{"1min":15.968999999999999,"5min":14.984,"15min":22.007999999999999},"min":{"1min":10.695,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":18.954999999999998,"5min":26.942,"15min":173.17599999999999},"last":14.446999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:17:02 2023","interfaces":[{"interface":"back","average":{"1min":14.409000000000001,"5min":14.747999999999999,"15min":27.568000000000001},"min":{"1min":10.997,"5min":9.3789999999999996,"15min":9.3789999999999996},"max":{"1min":17.913,"5min":19.596,"15min":350.50999999999999},"last":94.143000000000001},{"interface":"front","average":{"1min":16.77,"5min":16.920000000000002,"15min":24.178000000000001},"min":{"1min":12.005000000000001,"5min":10.826000000000001,"15min":8.3659999999999997},"max":{"1min":24.143000000000001,"5min":24.143000000000001,"15min":304.64499999999998},"last":90.555999999999997}]}]},{"osd":7,"up_from":13,"seq":55834575232,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70292,"kb_used_data":6948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301548,"statfs":{"total":96636764160,"available":96564785152,"internally_reserved":0,"allocated":7114752,"data_stored":3426237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.7469999999999999,"5min":9.6129999999999995,"15min":18.939},"min":{"1min":5.4649999999999999,"5min":5.2800000000000002,"15min":4.2489999999999997},"max":{"1min":13.460000000000001,"5min":17.106999999999999,"15min":259.01499999999999},"last":10.769},{"interface":"front","average":{"1min":14.004,"5min":14.007999999999999,"15min":23.878},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.016},"max":{"1min":22.023,"5min":22.023,"15min":292.73899999999998},"last":16.266999999999999}]},{"osd":1,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.0190000000000001,"5min":9.6690000000000005,"15min":19.279},"min":{"1min":4.5,"5min":4.5,"15min":4.5},"max":{"1min":17.582999999999998,"5min":20.797000000000001,"15min":414.57100000000003},"last":10.340999999999999},{"interface":"front","average":{"1min":10.18,"5min":10.154999999999999,"15min":17.631},"min":{"1min":6.9130000000000003,"5min":6.4880000000000004,"15min":5.7709999999999999},"max":{"1min":16.693999999999999,"5min":19.157,"15min":333.53800000000001},"last":9.2669999999999995}]},{"osd":2,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":11.805,"5min":12.065,"15min":21.433},"min":{"1min":7.7130000000000001,"5min":6.75,"15min":6.75},"max":{"1min":18.497,"5min":19.593,"15min":286.63499999999999},"last":11.417999999999999},{"interface":"front","average":{"1min":14.807,"5min":12.464,"15min":21.170000000000002},"min":{"1min":9.2910000000000004,"5min":7.0540000000000003,"15min":6.1280000000000001},"max":{"1min":19.707999999999998,"5min":21.053000000000001,"15min":299.54300000000001},"last":15.644}]},{"osd":3,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.760999999999999,"15min":24.027000000000001},"min":{"1min":8.9309999999999992,"5min":7.0810000000000004,"15min":6.6289999999999996},"max":{"1min":16.792000000000002,"5min":22.248000000000001,"15min":391.23200000000003},"last":14.438000000000001},{"interface":"front","average":{"1min":16.058,"5min":16.571000000000002,"15min":26.021999999999998},"min":{"1min":10.843,"5min":10.125,"15min":8.3949999999999996},"max":{"1min":20.483000000000001,"5min":22.477,"15min":362.20299999999997},"last":16.925999999999998}]},{"osd":4,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.838999999999999,"5min":17.010999999999999,"15min":23.010000000000002},"min":{"1min":12.465999999999999,"5min":7.6710000000000003,"15min":7.6710000000000003},"max":{"1min":20.530000000000001,"5min":22.053000000000001,"15min":293.358},"last":15.022},{"interface":"front","average":{"1min":14.430999999999999,"5min":13.443,"15min":20.056999999999999},"min":{"1min":9.4139999999999997,"5min":7.7149999999999999,"15min":6.6769999999999996},"max":{"1min":19.288,"5min":19.962,"15min":275.31799999999998},"last":17.555}]},{"osd":5,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":17.608000000000001,"5min":17.893000000000001,"15min":26.475000000000001},"min":{"1min":11.775,"5min":9.6720000000000006,"15min":7.7380000000000004},"max":{"1min":21.373000000000001,"5min":23.498000000000001,"15min":294.56400000000002},"last":18.363},{"interface":"front","average":{"1min":9.9760000000000009,"5min":11.784000000000001,"15min":22.891999999999999},"min":{"1min":6.3739999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"max":{"1min":17.087,"5min":21.634,"15min":280.83600000000001},"last":13.685}]},{"osd":6,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.66,"5min":15.532999999999999,"15min":24.460000000000001},"min":{"1min":11.808999999999999,"5min":10.372,"15min":8.5579999999999998},"max":{"1min":21.218,"5min":22.132000000000001,"15min":289.798},"last":19.135999999999999},{"interface":"front","average":{"1min":12.567,"5min":13.994999999999999,"15min":23.161999999999999},"min":{"1min":8.3300000000000001,"5min":7.5129999999999999,"15min":7.5129999999999999},"max":{"1min":18.094000000000001,"5min":20.853999999999999,"15min":303.30900000000003},"last":13.026}]}]},{"osd":6,"up_from":13,"seq":55834575231,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86752,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94285088,"statfs":{"total":96636764160,"available":96547930112,"internally_reserved":0,"allocated":6684672,"data_stored":3185909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.628,"5min":10.327,"15min":14.347},"min":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":4.1920000000000002},"max":{"1min":15.675000000000001,"5min":18.510000000000002,"15min":168.34100000000001},"last":6.4409999999999998},{"interface":"front","average":{"1min":9.1110000000000007,"5min":8.9320000000000004,"15min":13.920999999999999},"min":{"1min":5.6600000000000001,"5min":5.2050000000000001,"15min":5.2050000000000001},"max":{"1min":15.646000000000001,"5min":19.010999999999999,"15min":172.452},"last":6.093}]},{"osd":1,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":12.702999999999999,"5min":13.539,"15min":16.152000000000001},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.1870000000000003},"max":{"1min":18.241,"5min":20.835000000000001,"15min":206.43899999999999},"last":11.98},{"interface":"front","average":{"1min":16.012,"5min":16.016999999999999,"15min":19.134},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":7.2080000000000002},"max":{"1min":35.750999999999998,"5min":35.750999999999998,"15min":155.94},"last":15.548999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:17:05 2023","interfaces":[{"interface":"back","average":{"1min":11.721,"5min":11.587999999999999,"15min":16.103000000000002},"min":{"1min":8.5380000000000003,"5min":6.508,"15min":5.8940000000000001},"max":{"1min":20.643000000000001,"5min":20.643000000000001,"15min":168.23599999999999},"last":11.254},{"interface":"front","average":{"1min":10.148,"5min":10.307,"15min":15.826000000000001},"min":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"max":{"1min":13.957000000000001,"5min":18.495999999999999,"15min":352.52800000000002},"last":7.1609999999999996}]},{"osd":3,"last update":"Fri Dec 8 00:17:39 2023","interfaces":[{"interface":"back","average":{"1min":11.417999999999999,"5min":11.669,"15min":17.757999999999999},"min":{"1min":6.867,"5min":6.867,"15min":5.8789999999999996},"max":{"1min":21.459,"5min":21.459,"15min":286.92399999999998},"last":10.349},{"interface":"front","average":{"1min":17.001000000000001,"5min":15.720000000000001,"15min":22.109000000000002},"min":{"1min":9.8889999999999993,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":36.771000000000001,"5min":36.771000000000001,"15min":265.44400000000002},"last":16.123000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:46 2023","interfaces":[{"interface":"back","average":{"1min":13.489000000000001,"5min":13.015000000000001,"15min":18.609000000000002},"min":{"1min":7.3319999999999999,"5min":6.7389999999999999,"15min":6.7389999999999999},"max":{"1min":33.700000000000003,"5min":33.700000000000003,"15min":353.452},"last":11.003},{"interface":"front","average":{"1min":17.852,"5min":16.946000000000002,"15min":23.024000000000001},"min":{"1min":9.7469999999999999,"5min":9.7469999999999999,"15min":8.8499999999999996},"max":{"1min":37.723999999999997,"5min":37.723999999999997,"15min":341.87299999999999},"last":14.942}]},{"osd":5,"last update":"Fri Dec 8 00:17:05 2023","interfaces":[{"interface":"back","average":{"1min":19.035,"5min":18.305,"15min":24.193999999999999},"min":{"1min":11.776999999999999,"5min":11.776999999999999,"15min":10.817},"max":{"1min":34.421999999999997,"5min":34.421999999999997,"15min":158.316},"last":18.122},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.954000000000001,"15min":17.834},"min":{"1min":7.8220000000000001,"5min":7.1349999999999998,"15min":6.056},"max":{"1min":19.646999999999998,"5min":20.881,"15min":157.48500000000001},"last":14.061}]},{"osd":7,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":13.949999999999999,"5min":14.15,"15min":18.186},"min":{"1min":8.5719999999999992,"5min":8.5719999999999992,"15min":8.4930000000000003},"max":{"1min":30.861000000000001,"5min":30.861000000000001,"15min":152.49600000000001},"last":12.648},{"interface":"front","average":{"1min":18.798999999999999,"5min":18.960000000000001,"15min":23.832999999999998},"min":{"1min":11.927,"5min":11.927,"15min":11.388},"max":{"1min":38.762,"5min":38.762,"15min":299.93099999999998},"last":16.920999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575230,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75084,"kb_used_data":7348,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296756,"statfs":{"total":96636764160,"available":96559878144,"internally_reserved":0,"allocated":7524352,"data_stored":3824333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":7.3730000000000002,"5min":8.1890000000000001,"15min":14.359999999999999},"min":{"1min":4.226,"5min":4.226,"15min":4.226},"max":{"1min":15.23,"5min":17.338000000000001,"15min":401.80099999999999},"last":4.8259999999999996},{"interface":"front","average":{"1min":10.145,"5min":10.576000000000001,"15min":15.991},"min":{"1min":6.7610000000000001,"5min":4.9820000000000002,"15min":4.7690000000000001},"max":{"1min":14.962,"5min":17.664999999999999,"15min":175.446},"last":11.92}]},{"osd":1,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":12.279999999999999,"5min":12.211,"15min":17.137},"min":{"1min":7.9429999999999996,"5min":7.2839999999999998,"15min":5.5540000000000003},"max":{"1min":17.638000000000002,"5min":20.954999999999998,"15min":168.71299999999999},"last":12.691000000000001},{"interface":"front","average":{"1min":16.099,"5min":16.053000000000001,"15min":23.399000000000001},"min":{"1min":11.34,"5min":10.079000000000001,"15min":10.079000000000001},"max":{"1min":21.206,"5min":23.628,"15min":297.87299999999999},"last":13.789}]},{"osd":2,"last update":"Fri Dec 8 00:17:34 2023","interfaces":[{"interface":"back","average":{"1min":10.785,"5min":11.074999999999999,"15min":16.484999999999999},"min":{"1min":5.5460000000000003,"5min":5.5460000000000003,"15min":4.6980000000000004},"max":{"1min":17.513000000000002,"5min":20.484999999999999,"15min":222.297},"last":8.9390000000000001},{"interface":"front","average":{"1min":16.869,"5min":12.349,"15min":17.806999999999999},"min":{"1min":7.9729999999999999,"5min":6.0209999999999999,"15min":5.7789999999999999},"max":{"1min":88.489000000000004,"5min":88.489000000000004,"15min":402.50200000000001},"last":8.5470000000000006}]},{"osd":3,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":11.407999999999999,"5min":11.798999999999999,"15min":19.974},"min":{"1min":7.9189999999999996,"5min":6.6630000000000003,"15min":6.5190000000000001},"max":{"1min":16.321999999999999,"5min":18.933,"15min":295.20499999999998},"last":10.077999999999999},{"interface":"front","average":{"1min":14.73,"5min":15.452999999999999,"15min":22.516999999999999},"min":{"1min":9.6430000000000007,"5min":9.6280000000000001,"15min":8.4440000000000008},"max":{"1min":20.838999999999999,"5min":22.817,"15min":154.19200000000001},"last":13.223000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":17.074999999999999,"5min":17.097999999999999,"15min":23.731999999999999},"min":{"1min":10.194000000000001,"5min":9.8190000000000008,"15min":8.7309999999999999},"max":{"1min":24.096,"5min":24.096,"15min":337.97199999999998},"last":14.365},{"interface":"front","average":{"1min":13.545,"5min":14.003,"15min":19.593},"min":{"1min":11.345000000000001,"5min":9.0180000000000007,"15min":6.7350000000000003},"max":{"1min":16.024000000000001,"5min":22.512,"15min":169.846},"last":16.356999999999999}]},{"osd":6,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":15.096,"5min":15.509,"15min":19.300000000000001},"min":{"1min":11.891999999999999,"5min":8.4109999999999996,"15min":7.4109999999999996},"max":{"1min":17.960000000000001,"5min":24.210000000000001,"15min":169.93199999999999},"last":17.207999999999998},{"interface":"front","average":{"1min":17.221,"5min":17.995000000000001,"15min":24.332999999999998},"min":{"1min":12.741,"5min":10.512,"15min":8.7810000000000006},"max":{"1min":21.478000000000002,"5min":24.382999999999999,"15min":395.5},"last":14.941000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":12.48,"5min":12.404,"15min":23.420999999999999},"min":{"1min":7.8010000000000002,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":17.943999999999999,"5min":25.760999999999999,"15min":417.26299999999998},"last":17.952000000000002},{"interface":"front","average":{"1min":14.821,"5min":16.821000000000002,"15min":21.045999999999999},"min":{"1min":8.0950000000000006,"5min":8.0950000000000006,"15min":6.5780000000000003},"max":{"1min":19.294,"5min":26.504999999999999,"15min":352.34100000000001},"last":10.653}]}]},{"osd":4,"up_from":13,"seq":55834575230,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70080,"kb_used_data":6968,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301760,"statfs":{"total":96636764160,"available":96565002240,"internally_reserved":0,"allocated":7135232,"data_stored":3426877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.9109999999999996,"15min":15.105},"min":{"1min":4.9039999999999999,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":19.489999999999998,"5min":19.489999999999998,"15min":218.566},"last":6.6440000000000001},{"interface":"front","average":{"1min":9.6110000000000007,"5min":10.023,"15min":16.123999999999999},"min":{"1min":5.4039999999999999,"5min":5.1520000000000001,"15min":4.8300000000000001},"max":{"1min":19.678999999999998,"5min":19.678999999999998,"15min":299.358},"last":8.2959999999999994}]},{"osd":1,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":11.888999999999999,"15min":17.398},"min":{"1min":6.6440000000000001,"5min":6.0190000000000001,"15min":5.9249999999999998},"max":{"1min":16.940000000000001,"5min":18.978000000000002,"15min":291.13200000000001},"last":5.8319999999999999},{"interface":"front","average":{"1min":11.286,"5min":10.930999999999999,"15min":14.776},"min":{"1min":5.3490000000000002,"5min":5.3490000000000002,"15min":4.9509999999999996},"max":{"1min":16.937000000000001,"5min":18.469000000000001,"15min":147.71000000000001},"last":12.141999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":11.776999999999999,"15min":17.225000000000001},"min":{"1min":7.2880000000000003,"5min":6.8819999999999997,"15min":6.1539999999999999},"max":{"1min":86.319000000000003,"5min":86.319000000000003,"15min":300.22399999999999},"last":7.3710000000000004},{"interface":"front","average":{"1min":15.244,"5min":14.737,"15min":20.122},"min":{"1min":11.385,"5min":8.7119999999999997,"15min":6.7009999999999996},"max":{"1min":26.940999999999999,"5min":26.940999999999999,"15min":258.58499999999998},"last":11.417}]},{"osd":3,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.311,"5min":14.824999999999999,"15min":22.536999999999999},"min":{"1min":9.6920000000000002,"5min":7.4279999999999999,"15min":6.4219999999999997},"max":{"1min":51.329999999999998,"5min":51.329999999999998,"15min":400.41699999999997},"last":14.738},{"interface":"front","average":{"1min":15.747999999999999,"5min":13.279,"15min":19.422000000000001},"min":{"1min":9.8249999999999993,"5min":7.0460000000000003,"15min":7.0460000000000003},"max":{"1min":49.639000000000003,"5min":49.639000000000003,"15min":273.839},"last":12.786}]},{"osd":5,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.038,"5min":15.727,"15min":22.605},"min":{"1min":10.035,"5min":7.4939999999999998,"15min":6.7569999999999997},"max":{"1min":33.755000000000003,"5min":33.755000000000003,"15min":397.96199999999999},"last":16.248000000000001},{"interface":"front","average":{"1min":17.222999999999999,"5min":14.757,"15min":20.649000000000001},"min":{"1min":10.176,"5min":7.0129999999999999,"15min":6.9900000000000002},"max":{"1min":59.944000000000003,"5min":59.944000000000003,"15min":291.952},"last":14.034000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":19.376999999999999,"5min":15.308999999999999,"15min":20.122},"min":{"1min":8.5579999999999998,"5min":7.8940000000000001,"15min":6.1399999999999997},"max":{"1min":50.722999999999999,"5min":50.722999999999999,"15min":296.37},"last":10.465999999999999},{"interface":"front","average":{"1min":25.309000000000001,"5min":17.347000000000001,"15min":21.280000000000001},"min":{"1min":11.715,"5min":8.9130000000000003,"15min":7.8860000000000001},"max":{"1min":111.55,"5min":111.55,"15min":232.64099999999999},"last":15.648999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":17.195,"5min":16.981999999999999,"15min":22.972999999999999},"min":{"1min":11.882,"5min":9.6539999999999999,"15min":7.4279999999999999},"max":{"1min":22.263999999999999,"5min":22.263999999999999,"15min":235.828},"last":17.268000000000001},{"interface":"front","average":{"1min":16.57,"5min":16.664000000000001,"15min":24.167000000000002},"min":{"1min":11.945,"5min":9.6530000000000005,"15min":9.0869999999999997},"max":{"1min":19.780000000000001,"5min":22.067,"15min":298.74599999999998},"last":18.181999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T00:18:03.249 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.354+0000 7ffb84dc9640 10 monclient: get_monmap_and_config 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.354+0000 7ffb84dc9640 10 monclient: build_initial_monmap 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.354+0000 7ffb84dc9640 10 monclient: monmap: 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:03.355716+0000 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:03.355716+0000 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 1 Processor -- start 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 1 -- start start 2023-12-08T00:18:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient: init 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb8006c570 0x7ffb8006c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient(hunting): picked mon.noname-a con 0x7ffb8006c570 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb8013f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient(hunting): picked mon.noname-b con 0x7ffb8013f0c0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb801422b0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb80141e20 con 0x7ffb801422b0 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb8006ce90 con 0x7ffb8006c570 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8006bb70 con 0x7ffb8013f0c0 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.355+0000 7ffb84dc9640 10 monclient(hunting): authenticate will time out at 3283.543732s 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb7e59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb8006c570 0x7ffb8006c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb75d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb8013f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb7ed9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ffb801422b0 0x7ffb80142740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:57428/0 (socket says 172.21.15.129:57428) 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb7ed9d640 1 -- 172.21.15.129:0/4195705027 learned_addr learned my addr 172.21.15.129:0/4195705027 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb7e59c640 10 monclient(hunting): get_auth_request con 0x7ffb8006c570 auth_method 0 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb7e59c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb7e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb75d9b640 10 monclient(hunting): get_auth_request con 0x7ffb8013f0c0 auth_method 0 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb75d9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.356+0000 7ffb75d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.357+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.357+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.357+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.357+0000 7ffb75d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.357+0000 7ffb75d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.357+0000 7ffb75d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.358+0000 7ffb7dd9b640 1 -- 172.21.15.129:0/4195705027 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322319701 0 0) 0x7ffb80141e20 con 0x7ffb801422b0 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.358+0000 7ffb7dd9b640 10 monclient(hunting): _negotiate 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.358+0000 7ffb7dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.358+0000 7ffb7dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.358+0000 7ffb7dd9b640 10 monclient(hunting): my global_id is 20194 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.358+0000 7ffb7dd9b640 1 -- 172.21.15.129:0/4195705027 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c003520 con 0x7ffb801422b0 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_done global_id 15723 payload 274 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 1 -- 172.21.15.129:0/4195705027 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 msgr2=0x7ffb8013f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 1 --2- 172.21.15.129:0/4195705027 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb8013f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 1 -- 172.21.15.129:0/4195705027 >> v1:172.21.15.184:6789/0 conn(0x7ffb801422b0 legacy=0x7ffb80142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 10 monclient: _finish_hunting 0 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 1 monclient: found mon.noname-a 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.359+0000 7ffb7e59c640 1 -- 172.21.15.129:0/4195705027 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8013fa40 con 0x7ffb8006c570 2023-12-08T00:18:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.362+0000 7ffb7e59c640 1 --2- 172.21.15.129:0/4195705027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb8006c950 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7ffb700097d0 tx=0x7ffb7002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb3b391c0d91114 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb7dd9b640 1 -- 172.21.15.129:0/4195705027 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70031800 con 0x7ffb8006c570 2023-12-08T00:18:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb7dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb7dd9b640 10 monclient: dump: 2023-12-08T00:18:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb7dd9b640 10 monclient: _finish_auth 0 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb7dd9b640 10 monclient: _check_auth_tickets 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb84dc9640 5 monclient: authenticate success, global_id 15723 2023-12-08T00:18:03.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.364+0000 7ffb84dc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb7dd9b640 1 -- 172.21.15.129:0/4195705027 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb70031960 con 0x7ffb8006c570 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb7dd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb7dd9b640 1 -- 172.21.15.129:0/4195705027 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70031cd0 con 0x7ffb8006c570 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb7dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb7dd9b640 10 monclient: dump: 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.369 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb84dc9640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.365+0000 7ffb84dc9640 4 set_mon_vals no callback set 2023-12-08T00:18:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 10 monclient: shutdown 2023-12-08T00:18:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 -- 172.21.15.129:0/4195705027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 msgr2=0x7ffb8006c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/4195705027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb8006c950 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7ffb700097d0 tx=0x7ffb7002b8a0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 -- 172.21.15.129:0/4195705027 shutdown_connections 2023-12-08T00:18:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/4195705027 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb8013f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/4195705027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb8006c950 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 -- 172.21.15.129:0/4195705027 >> 172.21.15.129:0/4195705027 conn(0x7ffb80066b90 msgr2=0x7ffb800b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 -- 172.21.15.129:0/4195705027 shutdown_connections 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.366+0000 7ffb84dc9640 1 -- 172.21.15.129:0/4195705027 wait complete. 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 10 monclient: build_initial_monmap 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 10 monclient: monmap: 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:03.373 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 1 librados: starting msgr at 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 1 librados: starting objecter 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 1 Processor -- start 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 1 -- start start 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 1 librados: setting wanted keys 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 1 librados: calling monclient init 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.367+0000 7ffb84dc9640 10 monclient: init 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb800aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): picked mon.noname-a con 0x7ffb8006c570 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb80136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): picked mon.noname-c con 0x7ffb8013f0c0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb801422b0 0x7ffb80133a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): picked mon.noname-b con 0x7ffb801422b0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80138af0 con 0x7ffb8006c570 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801371d0 con 0x7ffb801422b0 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80137ae0 con 0x7ffb8013f0c0 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb84dc9640 10 monclient(hunting): authenticate will time out at 3283.556491s 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.368+0000 7ffb7e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb800aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb80136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb801422b0 0x7ffb80133a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb800aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58792/0 (socket says 172.21.15.129:58792) 2023-12-08T00:18:03.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 1 -- 172.21.15.129:0/1443045517 learned_addr learned my addr 172.21.15.129:0/1443045517 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 10 monclient(hunting): get_auth_request con 0x7ffb8006c570 auth_method 0 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 10 monclient(hunting): get_auth_request con 0x7ffb8013f0c0 auth_method 0 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 10 monclient(hunting): get_auth_request con 0x7ffb801422b0 auth_method 0 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.369+0000 7ffb75d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 10 monclient(hunting): handle_auth_done global_id 15726 payload 824 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 1 -- 172.21.15.129:0/1443045517 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 msgr2=0x7ffb80136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb80136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 1 -- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb801422b0 msgr2=0x7ffb80133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb801422b0 0x7ffb80133a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 10 monclient: _finish_hunting 0 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 1 monclient: found mon.noname-a 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:03.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:03.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7e59c640 1 -- 172.21.15.129:0/1443045517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb70009400 con 0x7ffb8006c570 2023-12-08T00:18:03.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.372+0000 7ffb7ed9d640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb801422b0 0x7ffb80133a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-08T00:18:03.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.375+0000 7ffb7e59c640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb800aef80 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7ffb7002b8e0 tx=0x7ffb700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:03.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.375+0000 7ffb777fe640 10 client.?.objecter ms_handle_connect 0x7ffb8006c570 2023-12-08T00:18:03.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.375+0000 7ffb777fe640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 1 -- 172.21.15.129:0/1443045517 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70003610 con 0x7ffb8006c570 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: dump: 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 1 -- 172.21.15.129:0/1443045517 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb70003770 con 0x7ffb8006c570 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 1 -- 172.21.15.129:0/1443045517 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70003b00 con 0x7ffb8006c570 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 5 monclient: authenticate success, global_id 15726 2023-12-08T00:18:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 10 monclient: _renew_subs 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb80136340 con 0x7ffb8006c570 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb7d59a640 4 set_mon_vals no callback set 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb777fe640 10 monclient: dump: 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.383 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 10 client.15726.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 10 monclient: _renew_subs 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb80134170 con 0x7ffb8006c570 2023-12-08T00:18:03.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.378+0000 7ffb84dc9640 1 librados: init done 2023-12-08T00:18:03.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.379+0000 7ffb84dc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:03.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.379+0000 7ffb84dc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:03.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.379+0000 7ffb84dc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:03.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.379+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8014c6f0 con 0x7ffb8006c570 2023-12-08T00:18:03.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.398+0000 7ffb777fe640 1 -- 172.21.15.129:0/1443045517 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffb70003c60 con 0x7ffb8006c570 2023-12-08T00:18:03.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.398+0000 7ffb777fe640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb6006c820 0x7ffb6006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.399+0000 7ffb75d9b640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb6006c820 0x7ffb6006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.399+0000 7ffb75d9b640 10 monclient: get_auth_request con 0x7ffb6006c820 auth_method 0 2023-12-08T00:18:03.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.399+0000 7ffb75d9b640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb6006c820 0x7ffb6006ece0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffb801d8970 tx=0x7ffb68009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:03.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.399+0000 7ffb777fe640 10 client.15726.objecter ms_handle_connect 0x7ffb6006c820 2023-12-08T00:18:03.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.402+0000 7ffb777fe640 1 -- 172.21.15.129:0/1443045517 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1046..1046 src has 978..1046) v4 ==== 6421+0+0 (secure 0 0 0) 0x7ffb7002e070 con 0x7ffb8006c570 2023-12-08T00:18:03.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.402+0000 7ffb777fe640 10 client.15726.objecter ms_dispatch 0x7ffb80000b90 osd_map(1046..1046 src has 978..1046) v4 2023-12-08T00:18:03.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.402+0000 7ffb777fe640 3 client.15726.objecter handle_osd_map got epochs [1046,1046] > 0 2023-12-08T00:18:03.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.402+0000 7ffb777fe640 3 client.15726.objecter handle_osd_map decoding full epoch 1046 2023-12-08T00:18:03.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.402+0000 7ffb777fe640 20 client.15726.objecter dump_active .. 0 homeless 2023-12-08T00:18:03.637 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.635+0000 7ffb777fe640 1 -- 172.21.15.129:0/1443045517 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb700767d0 con 0x7ffb8006c570 2023-12-08T00:18:03.637 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.635+0000 7ffb777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:03.638 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.635+0000 7ffb777fe640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.807+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb8006b3a0 con 0x7ffb6006c820 2023-12-08T00:18:03.815 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.813+0000 7ffb777fe640 1 -- 172.21.15.129:0/1443045517 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151287 (secure 0 0 0) 0x7ffb8006b3a0 con 0x7ffb6006c820 2023-12-08T00:18:03.816 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-08T00:18:03.817 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-08T00:18:03.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 10 librados: watch_flush enter 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 10 librados: watch_flush exit 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 20 client.15726.objecter shutdown clearing up homeless session... 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 10 client.15726.objecter successfully canceled tick 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb6006c820 msgr2=0x7ffb6006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb6006c820 0x7ffb6006ece0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffb801d8970 tx=0x7ffb68009190 comp rx=0 tx=0).stop 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 10 monclient: shutdown 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 msgr2=0x7ffb800aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb800aef80 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7ffb7002b8e0 tx=0x7ffb700046c0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 shutdown_connections 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7ffb6006c820 0x7ffb6006ece0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb801422b0 0x7ffb80133a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffb8013f0c0 0x7ffb80136bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 --2- 172.21.15.129:0/1443045517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb8006c570 0x7ffb800aef80 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 >> 172.21.15.129:0/1443045517 conn(0x7ffb80066b90 msgr2=0x7ffb8013c050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:03.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 shutdown_connections 2023-12-08T00:18:03.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 -- 172.21.15.129:0/1443045517 wait complete. 2023-12-08T00:18:03.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.815+0000 7ffb84dc9640 1 librados: shutdown 2023-12-08T00:18:03.828 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":2010,"stamp":"2023-12-08T00:18:02.135927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":97,"num_write":134,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":595240,"kb_used_data":54948,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754379480,"statfs":{"total":773094113280,"available":772484587520,"internally_reserved":0,"allocated":56266752,"data_stored":27347627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1792,"num_objects":-14,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14,"num_whiteouts":0,"num_read":-12,"num_read_kb":0,"num_write":-14,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000844"},"pg_stats":[{"pgid":"15.1f","version":"0'0","reported_seq":997,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.672927+0000","last_change":"2023-12-07T23:47:23.193178+0000","last_active":"2023-12-08T00:17:44.672927+0000","last_peered":"2023-12-08T00:17:44.672927+0000","last_clean":"2023-12-08T00:17:44.672927+0000","last_became_active":"2023-12-07T23:47:23.187382+0000","last_became_peered":"2023-12-07T23:47:23.187382+0000","last_unstale":"2023-12-08T00:17:44.672927+0000","last_undegraded":"2023-12-08T00:17:44.672927+0000","last_fullsized":"2023-12-08T00:17:44.672927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:46:32.619584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":990,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.069369+0000","last_change":"2023-12-07T23:47:26.349171+0000","last_active":"2023-12-08T00:17:45.069369+0000","last_peered":"2023-12-08T00:17:45.069369+0000","last_clean":"2023-12-08T00:17:45.069369+0000","last_became_active":"2023-12-07T23:47:26.088721+0000","last_became_peered":"2023-12-07T23:47:26.088721+0000","last_unstale":"2023-12-08T00:17:45.069369+0000","last_undegraded":"2023-12-08T00:17:45.069369+0000","last_fullsized":"2023-12-08T00:17:45.069369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:46:10.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1010,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.029374+0000","last_change":"2023-12-07T23:47:18.674241+0000","last_active":"2023-12-08T00:17:45.029374+0000","last_peered":"2023-12-08T00:17:45.029374+0000","last_clean":"2023-12-08T00:17:45.029374+0000","last_became_active":"2023-12-07T23:47:18.658074+0000","last_became_peered":"2023-12-07T23:47:18.658074+0000","last_unstale":"2023-12-08T00:17:45.029374+0000","last_undegraded":"2023-12-08T00:17:45.029374+0000","last_fullsized":"2023-12-08T00:17:45.029374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T00:31:54.317455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":987,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.798505+0000","last_change":"2023-12-07T23:47:39.142806+0000","last_active":"2023-12-08T00:18:00.798505+0000","last_peered":"2023-12-08T00:18:00.798505+0000","last_clean":"2023-12-08T00:18:00.798505+0000","last_became_active":"2023-12-07T23:47:39.133797+0000","last_became_peered":"2023-12-07T23:47:39.133797+0000","last_unstale":"2023-12-08T00:18:00.798505+0000","last_undegraded":"2023-12-08T00:18:00.798505+0000","last_fullsized":"2023-12-08T00:18:00.798505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:38:14.932982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":991,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.056223+0000","last_change":"2023-12-07T23:47:20.151369+0000","last_active":"2023-12-08T00:17:45.056223+0000","last_peered":"2023-12-08T00:17:45.056223+0000","last_clean":"2023-12-08T00:17:45.056223+0000","last_became_active":"2023-12-07T23:47:20.132690+0000","last_became_peered":"2023-12-07T23:47:20.132690+0000","last_unstale":"2023-12-08T00:17:45.056223+0000","last_undegraded":"2023-12-08T00:17:45.056223+0000","last_fullsized":"2023-12-08T00:17:45.056223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:51:11.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.029025+0000","last_change":"2023-12-07T23:47:19.053664+0000","last_active":"2023-12-08T00:17:45.029025+0000","last_peered":"2023-12-08T00:17:45.029025+0000","last_clean":"2023-12-08T00:17:45.029025+0000","last_became_active":"2023-12-07T23:47:18.926939+0000","last_became_peered":"2023-12-07T23:47:18.926939+0000","last_unstale":"2023-12-08T00:17:45.029025+0000","last_undegraded":"2023-12-08T00:17:45.029025+0000","last_fullsized":"2023-12-08T00:17:45.029025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:28:02.327836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1","version":"64'7","reported_seq":1006,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.725038+0000","last_change":"2023-12-07T23:47:15.440534+0000","last_active":"2023-12-08T00:18:00.725038+0000","last_peered":"2023-12-08T00:18:00.725038+0000","last_clean":"2023-12-08T00:18:00.725038+0000","last_became_active":"2023-12-07T23:47:15.386185+0000","last_became_peered":"2023-12-07T23:47:15.386185+0000","last_unstale":"2023-12-08T00:18:00.725038+0000","last_undegraded":"2023-12-08T00:18:00.725038+0000","last_fullsized":"2023-12-08T00:18:00.725038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:40:51.648331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":990,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.016890+0000","last_change":"2023-12-07T23:47:18.719911+0000","last_active":"2023-12-08T00:17:45.016890+0000","last_peered":"2023-12-08T00:17:45.016890+0000","last_clean":"2023-12-08T00:17:45.016890+0000","last_became_active":"2023-12-07T23:47:18.700591+0000","last_became_peered":"2023-12-07T23:47:18.700591+0000","last_unstale":"2023-12-08T00:17:45.016890+0000","last_undegraded":"2023-12-08T00:17:45.016890+0000","last_fullsized":"2023-12-08T00:17:45.016890+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:15:06.752653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.078513+0000","last_change":"2023-12-07T23:47:16.106412+0000","last_active":"2023-12-08T00:17:45.078513+0000","last_peered":"2023-12-08T00:17:45.078513+0000","last_clean":"2023-12-08T00:17:45.078513+0000","last_became_active":"2023-12-07T23:47:16.101044+0000","last_became_peered":"2023-12-07T23:47:16.101044+0000","last_unstale":"2023-12-08T00:17:45.078513+0000","last_undegraded":"2023-12-08T00:17:45.078513+0000","last_fullsized":"2023-12-08T00:17:45.078513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:51:45.972358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.067819+0000","last_change":"2023-12-07T23:47:19.725523+0000","last_active":"2023-12-08T00:17:45.067819+0000","last_peered":"2023-12-08T00:17:45.067819+0000","last_clean":"2023-12-08T00:17:45.067819+0000","last_became_active":"2023-12-07T23:47:18.123992+0000","last_became_peered":"2023-12-07T23:47:18.123992+0000","last_unstale":"2023-12-08T00:17:45.067819+0000","last_undegraded":"2023-12-08T00:17:45.067819+0000","last_fullsized":"2023-12-08T00:17:45.067819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:05:44.580089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":991,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.071367+0000","last_change":"2023-12-07T23:47:19.649219+0000","last_active":"2023-12-08T00:17:45.071367+0000","last_peered":"2023-12-08T00:17:45.071367+0000","last_clean":"2023-12-08T00:17:45.071367+0000","last_became_active":"2023-12-07T23:47:19.507339+0000","last_became_peered":"2023-12-07T23:47:19.507339+0000","last_unstale":"2023-12-08T00:17:45.071367+0000","last_undegraded":"2023-12-08T00:17:45.071367+0000","last_fullsized":"2023-12-08T00:17:45.071367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:38:46.882596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1007,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.639644+0000","last_change":"2023-12-07T23:48:06.632530+0000","last_active":"2023-12-08T00:17:44.639644+0000","last_peered":"2023-12-08T00:17:44.639644+0000","last_clean":"2023-12-08T00:17:44.639644+0000","last_became_active":"2023-12-07T23:48:06.627236+0000","last_became_peered":"2023-12-07T23:48:06.627236+0000","last_unstale":"2023-12-08T00:17:44.639644+0000","last_undegraded":"2023-12-08T00:17:44.639644+0000","last_fullsized":"2023-12-08T00:17:44.639644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:43:12.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1006,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.644941+0000","last_change":"2023-12-07T23:47:18.049591+0000","last_active":"2023-12-08T00:17:44.644941+0000","last_peered":"2023-12-08T00:17:44.644941+0000","last_clean":"2023-12-08T00:17:44.644941+0000","last_became_active":"2023-12-07T23:47:18.028936+0000","last_became_peered":"2023-12-07T23:47:18.028936+0000","last_unstale":"2023-12-08T00:17:44.644941+0000","last_undegraded":"2023-12-08T00:17:44.644941+0000","last_fullsized":"2023-12-08T00:17:44.644941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:11:43.305002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"841'88","reported_seq":1119,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.733737+0000","last_change":"2023-12-07T23:46:05.583767+0000","last_active":"2023-12-08T00:18:00.733737+0000","last_peered":"2023-12-08T00:18:00.733737+0000","last_clean":"2023-12-08T00:18:00.733737+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-08T00:18:00.733737+0000","last_undegraded":"2023-12-08T00:18:00.733737+0000","last_fullsized":"2023-12-08T00:18:00.733737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_clean_scrub_stamp":"2023-12-07T23:46:02.581722+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:48:03.670610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.057811+0000","last_change":"2023-12-07T23:46:13.815701+0000","last_active":"2023-12-08T00:17:45.057811+0000","last_peered":"2023-12-08T00:17:45.057811+0000","last_clean":"2023-12-08T00:17:45.057811+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-08T00:17:45.057811+0000","last_undegraded":"2023-12-08T00:17:45.057811+0000","last_fullsized":"2023-12-08T00:17:45.057811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:13.731531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":992,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.756380+0000","last_change":"2023-12-07T23:47:18.113515+0000","last_active":"2023-12-08T00:18:00.756380+0000","last_peered":"2023-12-08T00:18:00.756380+0000","last_clean":"2023-12-08T00:18:00.756380+0000","last_became_active":"2023-12-07T23:47:18.104193+0000","last_became_peered":"2023-12-07T23:47:18.104193+0000","last_unstale":"2023-12-08T00:18:00.756380+0000","last_undegraded":"2023-12-08T00:18:00.756380+0000","last_fullsized":"2023-12-08T00:18:00.756380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:39:47.005386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.013885+0000","last_change":"2023-12-07T23:46:13.418137+0000","last_active":"2023-12-08T00:17:45.013885+0000","last_peered":"2023-12-08T00:17:45.013885+0000","last_clean":"2023-12-08T00:17:45.013885+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-08T00:17:45.013885+0000","last_undegraded":"2023-12-08T00:17:45.013885+0000","last_fullsized":"2023-12-08T00:17:45.013885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:51:13.255892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1006,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.741729+0000","last_change":"2023-12-07T23:47:25.704244+0000","last_active":"2023-12-08T00:18:00.741729+0000","last_peered":"2023-12-08T00:18:00.741729+0000","last_clean":"2023-12-08T00:18:00.741729+0000","last_became_active":"2023-12-07T23:47:25.699009+0000","last_became_peered":"2023-12-07T23:47:25.699009+0000","last_unstale":"2023-12-08T00:18:00.741729+0000","last_undegraded":"2023-12-08T00:18:00.741729+0000","last_fullsized":"2023-12-08T00:18:00.741729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:23:03.360563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1001,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.639059+0000","last_change":"2023-12-07T23:46:13.343801+0000","last_active":"2023-12-08T00:17:44.639059+0000","last_peered":"2023-12-08T00:17:44.639059+0000","last_clean":"2023-12-08T00:17:44.639059+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-08T00:17:44.639059+0000","last_undegraded":"2023-12-08T00:17:44.639059+0000","last_fullsized":"2023-12-08T00:17:44.639059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:42:20.201085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1005,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.762618+0000","last_change":"2023-12-07T23:47:26.181304+0000","last_active":"2023-12-08T00:18:00.762618+0000","last_peered":"2023-12-08T00:18:00.762618+0000","last_clean":"2023-12-08T00:18:00.762618+0000","last_became_active":"2023-12-07T23:47:26.173821+0000","last_became_peered":"2023-12-07T23:47:26.173821+0000","last_unstale":"2023-12-08T00:18:00.762618+0000","last_undegraded":"2023-12-08T00:18:00.762618+0000","last_fullsized":"2023-12-08T00:18:00.762618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T23:47:56.552622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.028661+0000","last_change":"2023-12-07T23:46:13.873757+0000","last_active":"2023-12-08T00:17:45.028661+0000","last_peered":"2023-12-08T00:17:45.028661+0000","last_clean":"2023-12-08T00:17:45.028661+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-08T00:17:45.028661+0000","last_undegraded":"2023-12-08T00:17:45.028661+0000","last_fullsized":"2023-12-08T00:17:45.028661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:40:06.824492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1007,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.727055+0000","last_change":"2023-12-07T23:47:46.300836+0000","last_active":"2023-12-08T00:18:00.727055+0000","last_peered":"2023-12-08T00:18:00.727055+0000","last_clean":"2023-12-08T00:18:00.727055+0000","last_became_active":"2023-12-07T23:47:46.295177+0000","last_became_peered":"2023-12-07T23:47:46.295177+0000","last_unstale":"2023-12-08T00:18:00.727055+0000","last_undegraded":"2023-12-08T00:18:00.727055+0000","last_fullsized":"2023-12-08T00:18:00.727055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:54:00.508066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":994,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.055514+0000","last_change":"2023-12-07T23:46:13.905749+0000","last_active":"2023-12-08T00:17:45.055514+0000","last_peered":"2023-12-08T00:17:45.055514+0000","last_clean":"2023-12-08T00:17:45.055514+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-08T00:17:45.055514+0000","last_undegraded":"2023-12-08T00:17:45.055514+0000","last_fullsized":"2023-12-08T00:17:45.055514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:45:13.859049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":989,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.747705+0000","last_change":"2023-12-07T23:47:18.243486+0000","last_active":"2023-12-08T00:18:00.747705+0000","last_peered":"2023-12-08T00:18:00.747705+0000","last_clean":"2023-12-08T00:18:00.747705+0000","last_became_active":"2023-12-07T23:47:18.206551+0000","last_became_peered":"2023-12-07T23:47:18.206551+0000","last_unstale":"2023-12-08T00:18:00.747705+0000","last_undegraded":"2023-12-08T00:18:00.747705+0000","last_fullsized":"2023-12-08T00:18:00.747705+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:33:27.429730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.312242+0000","last_change":"2023-12-07T23:46:13.447293+0000","last_active":"2023-12-08T00:17:45.312242+0000","last_peered":"2023-12-08T00:17:45.312242+0000","last_clean":"2023-12-08T00:17:45.312242+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-08T00:17:45.312242+0000","last_undegraded":"2023-12-08T00:17:45.312242+0000","last_fullsized":"2023-12-08T00:17:45.312242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:00.005960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":988,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.039062+0000","last_change":"2023-12-07T23:47:15.786542+0000","last_active":"2023-12-08T00:17:45.039062+0000","last_peered":"2023-12-08T00:17:45.039062+0000","last_clean":"2023-12-08T00:17:45.039062+0000","last_became_active":"2023-12-07T23:47:15.770063+0000","last_became_peered":"2023-12-07T23:47:15.770063+0000","last_unstale":"2023-12-08T00:17:45.039062+0000","last_undegraded":"2023-12-08T00:17:45.039062+0000","last_fullsized":"2023-12-08T00:17:45.039062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:18:15.429980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1011,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.647483+0000","last_change":"2023-12-07T23:46:13.656681+0000","last_active":"2023-12-08T00:17:44.647483+0000","last_peered":"2023-12-08T00:17:44.647483+0000","last_clean":"2023-12-08T00:17:44.647483+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-08T00:17:44.647483+0000","last_undegraded":"2023-12-08T00:17:44.647483+0000","last_fullsized":"2023-12-08T00:17:44.647483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:29.359895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":989,"reported_epoch":1042,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.676520+0000","last_change":"2023-12-07T23:47:15.701780+0000","last_active":"2023-12-08T00:17:45.676520+0000","last_peered":"2023-12-08T00:17:45.676520+0000","last_clean":"2023-12-08T00:17:45.676520+0000","last_became_active":"2023-12-07T23:47:15.693158+0000","last_became_peered":"2023-12-07T23:47:15.693158+0000","last_unstale":"2023-12-08T00:17:45.676520+0000","last_undegraded":"2023-12-08T00:17:45.676520+0000","last_fullsized":"2023-12-08T00:17:45.676520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:12.790526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":995,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.078677+0000","last_change":"2023-12-07T23:46:13.620414+0000","last_active":"2023-12-08T00:17:45.078677+0000","last_peered":"2023-12-08T00:17:45.078677+0000","last_clean":"2023-12-08T00:17:45.078677+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-08T00:17:45.078677+0000","last_undegraded":"2023-12-08T00:17:45.078677+0000","last_fullsized":"2023-12-08T00:17:45.078677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:44:19.783064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":995,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.626794+0000","last_change":"2023-12-07T23:47:21.241228+0000","last_active":"2023-12-08T00:17:44.626794+0000","last_peered":"2023-12-08T00:17:44.626794+0000","last_clean":"2023-12-08T00:17:44.626794+0000","last_became_active":"2023-12-07T23:47:21.229799+0000","last_became_peered":"2023-12-07T23:47:21.229799+0000","last_unstale":"2023-12-08T00:17:44.626794+0000","last_undegraded":"2023-12-08T00:17:44.626794+0000","last_fullsized":"2023-12-08T00:17:44.626794+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:30:12.922388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":991,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.722464+0000","last_change":"2023-12-07T23:47:36.622692+0000","last_active":"2023-12-08T00:18:00.722464+0000","last_peered":"2023-12-08T00:18:00.722464+0000","last_clean":"2023-12-08T00:18:00.722464+0000","last_became_active":"2023-12-07T23:47:36.616695+0000","last_became_peered":"2023-12-07T23:47:36.616695+0000","last_unstale":"2023-12-08T00:18:00.722464+0000","last_undegraded":"2023-12-08T00:18:00.722464+0000","last_fullsized":"2023-12-08T00:18:00.722464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:48:24.932348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":991,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.731548+0000","last_change":"2023-12-07T23:47:36.574414+0000","last_active":"2023-12-08T00:18:00.731548+0000","last_peered":"2023-12-08T00:18:00.731548+0000","last_clean":"2023-12-08T00:18:00.731548+0000","last_became_active":"2023-12-07T23:47:36.567130+0000","last_became_peered":"2023-12-07T23:47:36.567130+0000","last_unstale":"2023-12-08T00:18:00.731548+0000","last_undegraded":"2023-12-08T00:18:00.731548+0000","last_fullsized":"2023-12-08T00:18:00.731548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:53:21.050211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.054680+0000","last_change":"2023-12-07T23:47:37.286320+0000","last_active":"2023-12-08T00:17:45.054680+0000","last_peered":"2023-12-08T00:17:45.054680+0000","last_clean":"2023-12-08T00:17:45.054680+0000","last_became_active":"2023-12-07T23:47:37.279080+0000","last_became_peered":"2023-12-07T23:47:37.279080+0000","last_unstale":"2023-12-08T00:17:45.054680+0000","last_undegraded":"2023-12-08T00:17:45.054680+0000","last_fullsized":"2023-12-08T00:17:45.054680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:43:22.378569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":990,"reported_epoch":1042,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.866514+0000","last_change":"2023-12-07T23:47:21.394853+0000","last_active":"2023-12-08T00:17:45.866514+0000","last_peered":"2023-12-08T00:17:45.866514+0000","last_clean":"2023-12-08T00:17:45.866514+0000","last_became_active":"2023-12-07T23:47:21.389382+0000","last_became_peered":"2023-12-07T23:47:21.389382+0000","last_unstale":"2023-12-08T00:17:45.866514+0000","last_undegraded":"2023-12-08T00:17:45.866514+0000","last_fullsized":"2023-12-08T00:17:45.866514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:18.044930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1005,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.751316+0000","last_change":"2023-12-07T23:47:20.975428+0000","last_active":"2023-12-08T00:18:00.751316+0000","last_peered":"2023-12-08T00:18:00.751316+0000","last_clean":"2023-12-08T00:18:00.751316+0000","last_became_active":"2023-12-07T23:47:20.968485+0000","last_became_peered":"2023-12-07T23:47:20.968485+0000","last_unstale":"2023-12-08T00:18:00.751316+0000","last_undegraded":"2023-12-08T00:18:00.751316+0000","last_fullsized":"2023-12-08T00:18:00.751316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:03:02.615974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":987,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.786637+0000","last_change":"2023-12-07T23:47:21.404597+0000","last_active":"2023-12-08T00:18:00.786637+0000","last_peered":"2023-12-08T00:18:00.786637+0000","last_clean":"2023-12-08T00:18:00.786637+0000","last_became_active":"2023-12-07T23:47:21.395304+0000","last_became_peered":"2023-12-07T23:47:21.395304+0000","last_unstale":"2023-12-08T00:18:00.786637+0000","last_undegraded":"2023-12-08T00:18:00.786637+0000","last_fullsized":"2023-12-08T00:18:00.786637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:02:02.413893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1004,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.759930+0000","last_change":"2023-12-07T23:47:38.795823+0000","last_active":"2023-12-08T00:18:00.759930+0000","last_peered":"2023-12-08T00:18:00.759930+0000","last_clean":"2023-12-08T00:18:00.759930+0000","last_became_active":"2023-12-07T23:47:38.790323+0000","last_became_peered":"2023-12-07T23:47:38.790323+0000","last_unstale":"2023-12-08T00:18:00.759930+0000","last_undegraded":"2023-12-08T00:18:00.759930+0000","last_fullsized":"2023-12-08T00:18:00.759930+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:40:01.316349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":990,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.015499+0000","last_change":"2023-12-07T23:47:37.187628+0000","last_active":"2023-12-08T00:17:45.015499+0000","last_peered":"2023-12-08T00:17:45.015499+0000","last_clean":"2023-12-08T00:17:45.015499+0000","last_became_active":"2023-12-07T23:47:37.180489+0000","last_became_peered":"2023-12-07T23:47:37.180489+0000","last_unstale":"2023-12-08T00:17:45.015499+0000","last_undegraded":"2023-12-08T00:17:45.015499+0000","last_fullsized":"2023-12-08T00:17:45.015499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:28:21.180560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1006,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.617573+0000","last_change":"2023-12-07T23:48:09.211978+0000","last_active":"2023-12-08T00:17:44.617573+0000","last_peered":"2023-12-08T00:17:44.617573+0000","last_clean":"2023-12-08T00:17:44.617573+0000","last_became_active":"2023-12-07T23:48:09.186728+0000","last_became_peered":"2023-12-07T23:48:09.186728+0000","last_unstale":"2023-12-08T00:17:44.617573+0000","last_undegraded":"2023-12-08T00:17:44.617573+0000","last_fullsized":"2023-12-08T00:17:44.617573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:09:04.978015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":988,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.764294+0000","last_change":"2023-12-07T23:47:20.131095+0000","last_active":"2023-12-08T00:18:00.764294+0000","last_peered":"2023-12-08T00:18:00.764294+0000","last_clean":"2023-12-08T00:18:00.764294+0000","last_became_active":"2023-12-07T23:47:20.123312+0000","last_became_peered":"2023-12-07T23:47:20.123312+0000","last_unstale":"2023-12-08T00:18:00.764294+0000","last_undegraded":"2023-12-08T00:18:00.764294+0000","last_fullsized":"2023-12-08T00:18:00.764294+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:27:53.246959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1006,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:44.636698+0000","last_change":"2023-12-07T23:47:18.960597+0000","last_active":"2023-12-08T00:17:44.636698+0000","last_peered":"2023-12-08T00:17:44.636698+0000","last_clean":"2023-12-08T00:17:44.636698+0000","last_became_active":"2023-12-07T23:47:18.946564+0000","last_became_peered":"2023-12-07T23:47:18.946564+0000","last_unstale":"2023-12-08T00:17:44.636698+0000","last_undegraded":"2023-12-08T00:17:44.636698+0000","last_fullsized":"2023-12-08T00:17:44.636698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:55:18.226923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":2,"acting":2,"num_store_stats":2},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477119,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82556,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289284,"statfs":{"total":96636764160,"available":96552226816,"internally_reserved":0,"allocated":7131136,"data_stored":3426749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":8.3040000000000003,"5min":8.2520000000000007,"15min":15.449999999999999},"min":{"1min":4.4939999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"max":{"1min":20.571999999999999,"5min":20.571999999999999,"15min":272.77100000000002},"last":17.207999999999998},{"interface":"front","average":{"1min":11.718,"5min":10.836,"15min":18.363},"min":{"1min":5.5990000000000002,"5min":4.9900000000000002,"15min":4.9649999999999999},"max":{"1min":47.829999999999998,"5min":47.829999999999998,"15min":288.42000000000002},"last":17.963000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.33,"5min":11.922000000000001,"15min":21.003},"min":{"1min":6.2279999999999998,"5min":6.2279999999999998,"15min":5.7649999999999997},"max":{"1min":73.218999999999994,"5min":73.218999999999994,"15min":320.57900000000001},"last":10.895},{"interface":"front","average":{"1min":16.672000000000001,"5min":13.327,"15min":23.423999999999999},"min":{"1min":9.1400000000000006,"5min":6.7279999999999998,"15min":5.5670000000000002},"max":{"1min":91.882000000000005,"5min":91.882000000000005,"15min":461.51799999999997},"last":20.463000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:38 2023","interfaces":[{"interface":"back","average":{"1min":11.853,"5min":9.7810000000000006,"15min":15.744},"min":{"1min":7.343,"5min":6.0389999999999997,"15min":5.6959999999999997},"max":{"1min":66.941999999999993,"5min":66.941999999999993,"15min":246.47499999999999},"last":19.763999999999999},{"interface":"front","average":{"1min":17.085000000000001,"5min":14.343,"15min":22.997},"min":{"1min":8.7669999999999995,"5min":8.7279999999999998,"15min":7.3079999999999998},"max":{"1min":69.078999999999994,"5min":69.078999999999994,"15min":356.02100000000002},"last":21.995999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":17.209,"5min":14.24,"15min":19.495999999999999},"min":{"1min":7.7300000000000004,"5min":7.048,"15min":6.7690000000000001},"max":{"1min":107.04900000000001,"5min":107.04900000000001,"15min":180.76400000000001},"last":14.782999999999999},{"interface":"front","average":{"1min":16.361000000000001,"5min":14.583,"15min":22.693000000000001},"min":{"1min":10.125,"5min":6.5629999999999997,"15min":6.2990000000000004},"max":{"1min":62.020000000000003,"5min":62.020000000000003,"15min":223.91300000000001},"last":21.356999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.823,"5min":15.294,"15min":24.542000000000002},"min":{"1min":10.445,"5min":8.9920000000000009,"15min":7.2889999999999997},"max":{"1min":19.890000000000001,"5min":29.309999999999999,"15min":251.892},"last":18.597999999999999},{"interface":"front","average":{"1min":29.396999999999998,"5min":19.707000000000001,"15min":28.649999999999999},"min":{"1min":13.223000000000001,"5min":11.711,"15min":10.568},"max":{"1min":265.69799999999998,"5min":265.69799999999998,"15min":279.791},"last":23.388000000000002}]},{"osd":6,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":21.244,"5min":14.326000000000001,"15min":19.547000000000001},"min":{"1min":7.2850000000000001,"5min":7.2850000000000001,"15min":7.2850000000000001},"max":{"1min":214.774,"5min":214.774,"15min":214.774},"last":12.128},{"interface":"front","average":{"1min":26.649999999999999,"5min":17.835000000000001,"15min":24.134},"min":{"1min":12.226000000000001,"5min":10.428000000000001,"15min":8.1440000000000001},"max":{"1min":229.54300000000001,"5min":229.54300000000001,"15min":229.54300000000001},"last":13.499000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":16.917000000000002,"5min":17.433,"15min":28.428000000000001},"min":{"1min":8.923,"5min":8.923,"15min":8.923},"max":{"1min":21.625,"5min":27.763999999999999,"15min":281.31},"last":24.106000000000002},{"interface":"front","average":{"1min":14.558999999999999,"5min":14.391999999999999,"15min":23.718},"min":{"1min":9.9049999999999994,"5min":8.5600000000000005,"15min":7.649},"max":{"1min":23.234999999999999,"5min":23.234999999999999,"15min":255.72499999999999},"last":15.93}]}]},{"osd":2,"up_from":16,"seq":68719477120,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69880,"kb_used_data":6292,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301960,"statfs":{"total":96636764160,"available":96565207040,"internally_reserved":0,"allocated":6443008,"data_stored":3053431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5220000000000002,"5min":8.8870000000000005,"15min":15.034000000000001},"min":{"1min":4.8140000000000001,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":20.408000000000001,"5min":20.408000000000001,"15min":249.42699999999999},"last":15.491},{"interface":"front","average":{"1min":9.9410000000000007,"5min":9.4969999999999999,"15min":15.816000000000001},"min":{"1min":5.9480000000000004,"5min":5.8230000000000004,"15min":5.1310000000000002},"max":{"1min":19.068999999999999,"5min":19.068999999999999,"15min":241.45599999999999},"last":11.285}]},{"osd":1,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":12.634,"5min":11.613,"15min":19.722000000000001},"min":{"1min":8.1159999999999997,"5min":6.8179999999999996,"15min":6.766},"max":{"1min":19.122,"5min":19.122,"15min":352.21100000000001},"last":12.108000000000001},{"interface":"front","average":{"1min":11.063000000000001,"5min":10.92,"15min":17.779},"min":{"1min":8.3200000000000003,"5min":6.2720000000000002,"15min":5.9109999999999996},"max":{"1min":14.936,"5min":18.652000000000001,"15min":327.81099999999998},"last":13.92}]},{"osd":3,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.925000000000001,"5min":10.75,"15min":19.754000000000001},"min":{"1min":7.798,"5min":6.2770000000000001,"15min":6.1550000000000002},"max":{"1min":15.944000000000001,"5min":19.123999999999999,"15min":287.56299999999999},"last":8.1010000000000009},{"interface":"front","average":{"1min":14.613,"5min":13.928000000000001,"15min":21.443999999999999},"min":{"1min":8.9900000000000002,"5min":8.3059999999999992,"15min":7.149},"max":{"1min":17.920000000000002,"5min":20.739000000000001,"15min":255.334},"last":16.969000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":15.124000000000001,"5min":15.16,"15min":23.259},"min":{"1min":8.9789999999999992,"5min":8.9220000000000006,"15min":8.1539999999999999},"max":{"1min":19.739999999999998,"5min":20.571000000000002,"15min":301.12799999999999},"last":20.748999999999999},{"interface":"front","average":{"1min":12.436999999999999,"5min":12.302,"15min":20.571000000000002},"min":{"1min":6.7450000000000001,"5min":6.7450000000000001,"15min":6.2649999999999997},"max":{"1min":19.678999999999998,"5min":23.859999999999999,"15min":348.38099999999997},"last":10.226000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":16.494,"15min":24.294},"min":{"1min":14.236000000000001,"5min":9.0410000000000004,"15min":8.5470000000000006},"max":{"1min":20.163,"5min":22.094000000000001,"15min":289.18700000000001},"last":23.734000000000002},{"interface":"front","average":{"1min":14.500999999999999,"5min":14.137,"15min":20.013000000000002},"min":{"1min":9.3870000000000005,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":20.331,"5min":20.667000000000002,"15min":248.63499999999999},"last":21.385000000000002}]},{"osd":6,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":16.123000000000001,"5min":15.634,"15min":21.067},"min":{"1min":10.611000000000001,"5min":9.9610000000000003,"15min":7.4240000000000004},"max":{"1min":19.946000000000002,"5min":21.585000000000001,"15min":349.41399999999999},"last":41.552999999999997},{"interface":"front","average":{"1min":11.491,"5min":12.48,"15min":20.643999999999998},"min":{"1min":7.431,"5min":7.423,"15min":7.423},"max":{"1min":19.212,"5min":21.007000000000001,"15min":231.422},"last":34.932000000000002}]},{"osd":7,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":13.156000000000001,"5min":14.472,"15min":24.436},"min":{"1min":8.6010000000000009,"5min":8.6010000000000009,"15min":8.6010000000000009},"max":{"1min":20.013999999999999,"5min":20.428000000000001,"15min":350.52199999999999},"last":16.106999999999999},{"interface":"front","average":{"1min":16.969000000000001,"5min":17.178000000000001,"15min":24.536000000000001},"min":{"1min":11.411,"5min":10.608000000000001,"15min":10.475},"max":{"1min":20.597000000000001,"5min":22.510999999999999,"15min":235.19200000000001},"last":14.58}]}]},{"osd":0,"up_from":16,"seq":68719477119,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70076,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301764,"statfs":{"total":96636764160,"available":96565006336,"internally_reserved":0,"allocated":7131136,"data_stored":3426749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 8 00:17:29 2023","interfaces":[{"interface":"back","average":{"1min":10.356,"5min":9.7119999999999997,"15min":18.596},"min":{"1min":5.2789999999999999,"5min":4.827,"15min":4.5830000000000002},"max":{"1min":38.834000000000003,"5min":38.834000000000003,"15min":317.99599999999998},"last":14.888999999999999},{"interface":"front","average":{"1min":11.698,"5min":9.2210000000000001,"15min":17.318999999999999},"min":{"1min":6.6840000000000002,"5min":4.9139999999999997,"15min":4.9139999999999997},"max":{"1min":49.262,"5min":49.262,"15min":313.14400000000001},"last":7.2000000000000002}]},{"osd":2,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":30.468,"5min":16.053000000000001,"15min":20.640000000000001},"min":{"1min":8.9280000000000008,"5min":7.1600000000000001,"15min":6.4740000000000002},"max":{"1min":196.96700000000001,"5min":196.96700000000001,"15min":228.02500000000001},"last":11.413},{"interface":"front","average":{"1min":23.956,"5min":14.147,"15min":18.375},"min":{"1min":6.8819999999999997,"5min":5.7530000000000001,"15min":5.6059999999999999},"max":{"1min":168.91300000000001,"5min":168.91300000000001,"15min":197.78899999999999},"last":8.3989999999999991}]},{"osd":3,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":14.613,"5min":13.257,"15min":22.777999999999999},"min":{"1min":10.614000000000001,"5min":8.0500000000000007,"15min":7.0069999999999997},"max":{"1min":19.207999999999998,"5min":19.207999999999998,"15min":239.50999999999999},"last":12.526},{"interface":"front","average":{"1min":16.495999999999999,"5min":16.079000000000001,"15min":24.495999999999999},"min":{"1min":10.835000000000001,"5min":10.667,"15min":9.141},"max":{"1min":20.774000000000001,"5min":29.132999999999999,"15min":241.839},"last":15.701000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":8.8249999999999993,"5min":8.9369999999999994,"15min":14.958},"min":{"1min":6.4029999999999996,"5min":5.5570000000000004,"15min":5.5430000000000001},"max":{"1min":22.925000000000001,"5min":22.925000000000001,"15min":211.452},"last":10.794},{"interface":"front","average":{"1min":16.763999999999999,"5min":16.686,"15min":22.658000000000001},"min":{"1min":10.212999999999999,"5min":10.212999999999999,"15min":9.3030000000000008},"max":{"1min":24.173999999999999,"5min":29.969000000000001,"15min":212.459},"last":16.332000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":21.459,"5min":14.852,"15min":19.033999999999999},"min":{"1min":8.6750000000000007,"5min":7.8970000000000002,"15min":5.9740000000000002},"max":{"1min":175.68100000000001,"5min":175.68100000000001,"15min":175.68100000000001},"last":9.8079999999999998},{"interface":"front","average":{"1min":22.960000000000001,"5min":13.959,"15min":19.902000000000001},"min":{"1min":8.1950000000000003,"5min":6.7789999999999999,"15min":6.6660000000000004},"max":{"1min":186.625,"5min":186.625,"15min":186.625},"last":13.194000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":17.789999999999999,"15min":26.085000000000001},"min":{"1min":11.406000000000001,"5min":11.406000000000001,"15min":10.641999999999999},"max":{"1min":26.757999999999999,"5min":30.727,"15min":227.892},"last":17.099},{"interface":"front","average":{"1min":16.129999999999999,"5min":14.592000000000001,"15min":20.123999999999999},"min":{"1min":10.122,"5min":8.2919999999999998,"15min":8.2919999999999998},"max":{"1min":44.432000000000002,"5min":44.432000000000002,"15min":153.59700000000001},"last":14.284000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.475000000000001,"5min":16.189,"15min":24.940000000000001},"min":{"1min":11.638,"5min":9.3550000000000004,"15min":8.9369999999999994},"max":{"1min":39.835999999999999,"5min":39.835999999999999,"15min":264.44600000000003},"last":19.535},{"interface":"front","average":{"1min":14.044,"5min":12.993,"15min":20.210000000000001},"min":{"1min":7.8209999999999997,"5min":7.5350000000000001,"15min":7.5},"max":{"1min":41.091000000000001,"5min":41.091000000000001,"15min":243.84899999999999},"last":17.846}]}]},{"osd":1,"up_from":14,"seq":60129542525,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70520,"kb_used_data":6936,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301320,"statfs":{"total":96636764160,"available":96564551680,"internally_reserved":0,"allocated":7102464,"data_stored":3577342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.6829999999999998,"5min":9.3740000000000006,"15min":17.443000000000001},"min":{"1min":5.8540000000000001,"5min":5.6449999999999996,"15min":4.2539999999999996},"max":{"1min":17.231000000000002,"5min":18.986999999999998,"15min":324.46600000000001},"last":7.7610000000000001},{"interface":"front","average":{"1min":14.244,"5min":13.406000000000001,"15min":22.675000000000001},"min":{"1min":7.2469999999999999,"5min":6.4950000000000001,"15min":5.5369999999999999},"max":{"1min":21.832000000000001,"5min":21.832000000000001,"15min":342.39299999999997},"last":10.605}]},{"osd":2,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.9380000000000006,"5min":9.7550000000000008,"15min":20.015000000000001},"min":{"1min":5.6289999999999996,"5min":4.7409999999999997,"15min":4.7409999999999997},"max":{"1min":15.329000000000001,"5min":18.332000000000001,"15min":309.75999999999999},"last":83.418999999999997},{"interface":"front","average":{"1min":13.932,"5min":13.237,"15min":23.968},"min":{"1min":7.7919999999999998,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":19.77,"5min":22.640999999999998,"15min":343.49900000000002},"last":17.074999999999999}]},{"osd":3,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":13.263,"5min":14.33,"15min":22.765000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"max":{"1min":17.984999999999999,"5min":20.395,"15min":345.07600000000002},"last":11.33},{"interface":"front","average":{"1min":10.297000000000001,"5min":9.077,"15min":15.353},"min":{"1min":7.5209999999999999,"5min":6.1050000000000004,"15min":5.3090000000000002},"max":{"1min":15.089,"5min":16.981999999999999,"15min":174.20599999999999},"last":8.0690000000000008}]},{"osd":4,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.705,"5min":11.641999999999999,"15min":19.434999999999999},"min":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.0419999999999998},"max":{"1min":18.649999999999999,"5min":19.026,"15min":290.358},"last":12.401999999999999},{"interface":"front","average":{"1min":12.657999999999999,"5min":12.154999999999999,"15min":19.594999999999999},"min":{"1min":7.8029999999999999,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":17.945,"5min":19.114000000000001,"15min":203.929},"last":15.039999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":32.521999999999998,"5min":19.515000000000001,"15min":24.640999999999998},"min":{"1min":10.952999999999999,"5min":10.938000000000001,"15min":7.7690000000000001},"max":{"1min":333.11799999999999,"5min":333.11799999999999,"15min":371.50599999999997},"last":13.135999999999999},{"interface":"front","average":{"1min":28.724,"5min":15.845000000000001,"15min":19.704000000000001},"min":{"1min":9.0389999999999997,"5min":7.6150000000000002,"15min":7.282},"max":{"1min":298.12099999999998,"5min":298.12099999999998,"15min":298.12099999999998},"last":8.6370000000000005}]},{"osd":6,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":14.052,"5min":13.733000000000001,"15min":19.829000000000001},"min":{"1min":9.0879999999999992,"5min":7.3490000000000002,"15min":7.3490000000000002},"max":{"1min":19.085000000000001,"5min":24.93,"15min":292.72300000000001},"last":16.273},{"interface":"front","average":{"1min":15.968999999999999,"5min":14.984,"15min":22.007999999999999},"min":{"1min":10.695,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":18.954999999999998,"5min":26.942,"15min":173.17599999999999},"last":14.446999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:17:02 2023","interfaces":[{"interface":"back","average":{"1min":14.409000000000001,"5min":14.747999999999999,"15min":27.568000000000001},"min":{"1min":10.997,"5min":9.3789999999999996,"15min":9.3789999999999996},"max":{"1min":17.913,"5min":19.596,"15min":350.50999999999999},"last":94.143000000000001},{"interface":"front","average":{"1min":16.77,"5min":16.920000000000002,"15min":24.178000000000001},"min":{"1min":12.005000000000001,"5min":10.826000000000001,"15min":8.3659999999999997},"max":{"1min":24.143000000000001,"5min":24.143000000000001,"15min":304.64499999999998},"last":90.555999999999997}]}]},{"osd":7,"up_from":13,"seq":55834575232,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70292,"kb_used_data":6948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301548,"statfs":{"total":96636764160,"available":96564785152,"internally_reserved":0,"allocated":7114752,"data_stored":3426237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.7469999999999999,"5min":9.6129999999999995,"15min":18.939},"min":{"1min":5.4649999999999999,"5min":5.2800000000000002,"15min":4.2489999999999997},"max":{"1min":13.460000000000001,"5min":17.106999999999999,"15min":259.01499999999999},"last":10.769},{"interface":"front","average":{"1min":14.004,"5min":14.007999999999999,"15min":23.878},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.016},"max":{"1min":22.023,"5min":22.023,"15min":292.73899999999998},"last":16.266999999999999}]},{"osd":1,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.0190000000000001,"5min":9.6690000000000005,"15min":19.279},"min":{"1min":4.5,"5min":4.5,"15min":4.5},"max":{"1min":17.582999999999998,"5min":20.797000000000001,"15min":414.57100000000003},"last":10.340999999999999},{"interface":"front","average":{"1min":10.18,"5min":10.154999999999999,"15min":17.631},"min":{"1min":6.9130000000000003,"5min":6.4880000000000004,"15min":5.7709999999999999},"max":{"1min":16.693999999999999,"5min":19.157,"15min":333.53800000000001},"last":9.2669999999999995}]},{"osd":2,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":11.805,"5min":12.065,"15min":21.433},"min":{"1min":7.7130000000000001,"5min":6.75,"15min":6.75},"max":{"1min":18.497,"5min":19.593,"15min":286.63499999999999},"last":11.417999999999999},{"interface":"front","average":{"1min":14.807,"5min":12.464,"15min":21.170000000000002},"min":{"1min":9.2910000000000004,"5min":7.0540000000000003,"15min":6.1280000000000001},"max":{"1min":19.707999999999998,"5min":21.053000000000001,"15min":299.54300000000001},"last":15.644}]},{"osd":3,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.760999999999999,"15min":24.027000000000001},"min":{"1min":8.9309999999999992,"5min":7.0810000000000004,"15min":6.6289999999999996},"max":{"1min":16.792000000000002,"5min":22.248000000000001,"15min":391.23200000000003},"last":14.438000000000001},{"interface":"front","average":{"1min":16.058,"5min":16.571000000000002,"15min":26.021999999999998},"min":{"1min":10.843,"5min":10.125,"15min":8.3949999999999996},"max":{"1min":20.483000000000001,"5min":22.477,"15min":362.20299999999997},"last":16.925999999999998}]},{"osd":4,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.838999999999999,"5min":17.010999999999999,"15min":23.010000000000002},"min":{"1min":12.465999999999999,"5min":7.6710000000000003,"15min":7.6710000000000003},"max":{"1min":20.530000000000001,"5min":22.053000000000001,"15min":293.358},"last":15.022},{"interface":"front","average":{"1min":14.430999999999999,"5min":13.443,"15min":20.056999999999999},"min":{"1min":9.4139999999999997,"5min":7.7149999999999999,"15min":6.6769999999999996},"max":{"1min":19.288,"5min":19.962,"15min":275.31799999999998},"last":17.555}]},{"osd":5,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":17.608000000000001,"5min":17.893000000000001,"15min":26.475000000000001},"min":{"1min":11.775,"5min":9.6720000000000006,"15min":7.7380000000000004},"max":{"1min":21.373000000000001,"5min":23.498000000000001,"15min":294.56400000000002},"last":18.363},{"interface":"front","average":{"1min":9.9760000000000009,"5min":11.784000000000001,"15min":22.891999999999999},"min":{"1min":6.3739999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"max":{"1min":17.087,"5min":21.634,"15min":280.83600000000001},"last":13.685}]},{"osd":6,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.66,"5min":15.532999999999999,"15min":24.460000000000001},"min":{"1min":11.808999999999999,"5min":10.372,"15min":8.5579999999999998},"max":{"1min":21.218,"5min":22.132000000000001,"15min":289.798},"last":19.135999999999999},{"interface":"front","average":{"1min":12.567,"5min":13.994999999999999,"15min":23.161999999999999},"min":{"1min":8.3300000000000001,"5min":7.5129999999999999,"15min":7.5129999999999999},"max":{"1min":18.094000000000001,"5min":20.853999999999999,"15min":303.30900000000003},"last":13.026}]}]},{"osd":6,"up_from":13,"seq":55834575231,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86752,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94285088,"statfs":{"total":96636764160,"available":96547930112,"internally_reserved":0,"allocated":6684672,"data_stored":3185909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.628,"5min":10.327,"15min":14.347},"min":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":4.1920000000000002},"max":{"1min":15.675000000000001,"5min":18.510000000000002,"15min":168.34100000000001},"last":6.4409999999999998},{"interface":"front","average":{"1min":9.1110000000000007,"5min":8.9320000000000004,"15min":13.920999999999999},"min":{"1min":5.6600000000000001,"5min":5.2050000000000001,"15min":5.2050000000000001},"max":{"1min":15.646000000000001,"5min":19.010999999999999,"15min":172.452},"last":6.093}]},{"osd":1,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":12.702999999999999,"5min":13.539,"15min":16.152000000000001},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.1870000000000003},"max":{"1min":18.241,"5min":20.835000000000001,"15min":206.43899999999999},"last":11.98},{"interface":"front","average":{"1min":16.012,"5min":16.016999999999999,"15min":19.134},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":7.2080000000000002},"max":{"1min":35.750999999999998,"5min":35.750999999999998,"15min":155.94},"last":15.548999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:17:05 2023","interfaces":[{"interface":"back","average":{"1min":11.721,"5min":11.587999999999999,"15min":16.103000000000002},"min":{"1min":8.5380000000000003,"5min":6.508,"15min":5.8940000000000001},"max":{"1min":20.643000000000001,"5min":20.643000000000001,"15min":168.23599999999999},"last":11.254},{"interface":"front","average":{"1min":10.148,"5min":10.307,"15min":15.826000000000001},"min":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"max":{"1min":13.957000000000001,"5min":18.495999999999999,"15min":352.52800000000002},"last":7.1609999999999996}]},{"osd":3,"last update":"Fri Dec 8 00:17:39 2023","interfaces":[{"interface":"back","average":{"1min":11.417999999999999,"5min":11.669,"15min":17.757999999999999},"min":{"1min":6.867,"5min":6.867,"15min":5.8789999999999996},"max":{"1min":21.459,"5min":21.459,"15min":286.92399999999998},"last":10.349},{"interface":"front","average":{"1min":17.001000000000001,"5min":15.720000000000001,"15min":22.109000000000002},"min":{"1min":9.8889999999999993,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":36.771000000000001,"5min":36.771000000000001,"15min":265.44400000000002},"last":16.123000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:46 2023","interfaces":[{"interface":"back","average":{"1min":13.489000000000001,"5min":13.015000000000001,"15min":18.609000000000002},"min":{"1min":7.3319999999999999,"5min":6.7389999999999999,"15min":6.7389999999999999},"max":{"1min":33.700000000000003,"5min":33.700000000000003,"15min":353.452},"last":11.003},{"interface":"front","average":{"1min":17.852,"5min":16.946000000000002,"15min":23.024000000000001},"min":{"1min":9.7469999999999999,"5min":9.7469999999999999,"15min":8.8499999999999996},"max":{"1min":37.723999999999997,"5min":37.723999999999997,"15min":341.87299999999999},"last":14.942}]},{"osd":5,"last update":"Fri Dec 8 00:17:05 2023","interfaces":[{"interface":"back","average":{"1min":19.035,"5min":18.305,"15min":24.193999999999999},"min":{"1min":11.776999999999999,"5min":11.776999999999999,"15min":10.817},"max":{"1min":34.421999999999997,"5min":34.421999999999997,"15min":158.316},"last":18.122},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.954000000000001,"15min":17.834},"min":{"1min":7.8220000000000001,"5min":7.1349999999999998,"15min":6.056},"max":{"1min":19.646999999999998,"5min":20.881,"15min":157.48500000000001},"last":14.061}]},{"osd":7,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":13.949999999999999,"5min":14.15,"15min":18.186},"min":{"1min":8.5719999999999992,"5min":8.5719999999999992,"15min":8.4930000000000003},"max":{"1min":30.861000000000001,"5min":30.861000000000001,"15min":152.49600000000001},"last":12.648},{"interface":"front","average":{"1min":18.798999999999999,"5min":18.960000000000001,"15min":23.832999999999998},"min":{"1min":11.927,"5min":11.927,"15min":11.388},"max":{"1min":38.762,"5min":38.762,"15min":299.93099999999998},"last":16.920999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575230,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75084,"kb_used_data":7348,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296756,"statfs":{"total":96636764160,"available":96559878144,"internally_reserved":0,"allocated":7524352,"data_stored":3824333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":7.3730000000000002,"5min":8.1890000000000001,"15min":14.359999999999999},"min":{"1min":4.226,"5min":4.226,"15min":4.226},"max":{"1min":15.23,"5min":17.338000000000001,"15min":401.80099999999999},"last":4.8259999999999996},{"interface":"front","average":{"1min":10.145,"5min":10.576000000000001,"15min":15.991},"min":{"1min":6.7610000000000001,"5min":4.9820000000000002,"15min":4.7690000000000001},"max":{"1min":14.962,"5min":17.664999999999999,"15min":175.446},"last":11.92}]},{"osd":1,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":12.279999999999999,"5min":12.211,"15min":17.137},"min":{"1min":7.9429999999999996,"5min":7.2839999999999998,"15min":5.5540000000000003},"max":{"1min":17.638000000000002,"5min":20.954999999999998,"15min":168.71299999999999},"last":12.691000000000001},{"interface":"front","average":{"1min":16.099,"5min":16.053000000000001,"15min":23.399000000000001},"min":{"1min":11.34,"5min":10.079000000000001,"15min":10.079000000000001},"max":{"1min":21.206,"5min":23.628,"15min":297.87299999999999},"last":13.789}]},{"osd":2,"last update":"Fri Dec 8 00:17:34 2023","interfaces":[{"interface":"back","average":{"1min":10.785,"5min":11.074999999999999,"15min":16.484999999999999},"min":{"1min":5.5460000000000003,"5min":5.5460000000000003,"15min":4.6980000000000004},"max":{"1min":17.513000000000002,"5min":20.484999999999999,"15min":222.297},"last":8.9390000000000001},{"interface":"front","average":{"1min":16.869,"5min":12.349,"15min":17.806999999999999},"min":{"1min":7.9729999999999999,"5min":6.0209999999999999,"15min":5.7789999999999999},"max":{"1min":88.489000000000004,"5min":88.489000000000004,"15min":402.50200000000001},"last":8.5470000000000006}]},{"osd":3,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":11.407999999999999,"5min":11.798999999999999,"15min":19.974},"min":{"1min":7.9189999999999996,"5min":6.6630000000000003,"15min":6.5190000000000001},"max":{"1min":16.321999999999999,"5min":18.933,"15min":295.20499999999998},"last":10.077999999999999},{"interface":"front","average":{"1min":14.73,"5min":15.452999999999999,"15min":22.516999999999999},"min":{"1min":9.6430000000000007,"5min":9.6280000000000001,"15min":8.4440000000000008},"max":{"1min":20.838999999999999,"5min":22.817,"15min":154.19200000000001},"last":13.223000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":17.074999999999999,"5min":17.097999999999999,"15min":23.731999999999999},"min":{"1min":10.194000000000001,"5min":9.8190000000000008,"15min":8.7309999999999999},"max":{"1min":24.096,"5min":24.096,"15min":337.97199999999998},"last":14.365},{"interface":"front","average":{"1min":13.545,"5min":14.003,"15min":19.593},"min":{"1min":11.345000000000001,"5min":9.0180000000000007,"15min":6.7350000000000003},"max":{"1min":16.024000000000001,"5min":22.512,"15min":169.846},"last":16.356999999999999}]},{"osd":6,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":15.096,"5min":15.509,"15min":19.300000000000001},"min":{"1min":11.891999999999999,"5min":8.4109999999999996,"15min":7.4109999999999996},"max":{"1min":17.960000000000001,"5min":24.210000000000001,"15min":169.93199999999999},"last":17.207999999999998},{"interface":"front","average":{"1min":17.221,"5min":17.995000000000001,"15min":24.332999999999998},"min":{"1min":12.741,"5min":10.512,"15min":8.7810000000000006},"max":{"1min":21.478000000000002,"5min":24.382999999999999,"15min":395.5},"last":14.941000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:16:57 2023","interfaces":[{"interface":"back","average":{"1min":12.48,"5min":12.404,"15min":23.420999999999999},"min":{"1min":7.8010000000000002,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":17.943999999999999,"5min":25.760999999999999,"15min":417.26299999999998},"last":17.952000000000002},{"interface":"front","average":{"1min":14.821,"5min":16.821000000000002,"15min":21.045999999999999},"min":{"1min":8.0950000000000006,"5min":8.0950000000000006,"15min":6.5780000000000003},"max":{"1min":19.294,"5min":26.504999999999999,"15min":352.34100000000001},"last":10.653}]}]},{"osd":4,"up_from":13,"seq":55834575230,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70080,"kb_used_data":6968,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301760,"statfs":{"total":96636764160,"available":96565002240,"internally_reserved":0,"allocated":7135232,"data_stored":3426877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.9109999999999996,"15min":15.105},"min":{"1min":4.9039999999999999,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":19.489999999999998,"5min":19.489999999999998,"15min":218.566},"last":6.6440000000000001},{"interface":"front","average":{"1min":9.6110000000000007,"5min":10.023,"15min":16.123999999999999},"min":{"1min":5.4039999999999999,"5min":5.1520000000000001,"15min":4.8300000000000001},"max":{"1min":19.678999999999998,"5min":19.678999999999998,"15min":299.358},"last":8.2959999999999994}]},{"osd":1,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":11.888999999999999,"15min":17.398},"min":{"1min":6.6440000000000001,"5min":6.0190000000000001,"15min":5.9249999999999998},"max":{"1min":16.940000000000001,"5min":18.978000000000002,"15min":291.13200000000001},"last":5.8319999999999999},{"interface":"front","average":{"1min":11.286,"5min":10.930999999999999,"15min":14.776},"min":{"1min":5.3490000000000002,"5min":5.3490000000000002,"15min":4.9509999999999996},"max":{"1min":16.937000000000001,"5min":18.469000000000001,"15min":147.71000000000001},"last":12.141999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":11.776999999999999,"15min":17.225000000000001},"min":{"1min":7.2880000000000003,"5min":6.8819999999999997,"15min":6.1539999999999999},"max":{"1min":86.319000000000003,"5min":86.319000000000003,"15min":300.22399999999999},"last":7.3710000000000004},{"interface":"front","average":{"1min":15.244,"5min":14.737,"15min":20.122},"min":{"1min":11.385,"5min":8.7119999999999997,"15min":6.7009999999999996},"max":{"1min":26.940999999999999,"5min":26.940999999999999,"15min":258.58499999999998},"last":11.417}]},{"osd":3,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.311,"5min":14.824999999999999,"15min":22.536999999999999},"min":{"1min":9.6920000000000002,"5min":7.4279999999999999,"15min":6.4219999999999997},"max":{"1min":51.329999999999998,"5min":51.329999999999998,"15min":400.41699999999997},"last":14.738},{"interface":"front","average":{"1min":15.747999999999999,"5min":13.279,"15min":19.422000000000001},"min":{"1min":9.8249999999999993,"5min":7.0460000000000003,"15min":7.0460000000000003},"max":{"1min":49.639000000000003,"5min":49.639000000000003,"15min":273.839},"last":12.786}]},{"osd":5,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.038,"5min":15.727,"15min":22.605},"min":{"1min":10.035,"5min":7.4939999999999998,"15min":6.7569999999999997},"max":{"1min":33.755000000000003,"5min":33.755000000000003,"15min":397.96199999999999},"last":16.248000000000001},{"interface":"front","average":{"1min":17.222999999999999,"5min":14.757,"15min":20.649000000000001},"min":{"1min":10.176,"5min":7.0129999999999999,"15min":6.9900000000000002},"max":{"1min":59.944000000000003,"5min":59.944000000000003,"15min":291.952},"last":14.034000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":19.376999999999999,"5min":15.308999999999999,"15min":20.122},"min":{"1min":8.5579999999999998,"5min":7.8940000000000001,"15min":6.1399999999999997},"max":{"1min":50.722999999999999,"5min":50.722999999999999,"15min":296.37},"last":10.465999999999999},{"interface":"front","average":{"1min":25.309000000000001,"5min":17.347000000000001,"15min":21.280000000000001},"min":{"1min":11.715,"5min":8.9130000000000003,"15min":7.8860000000000001},"max":{"1min":111.55,"5min":111.55,"15min":232.64099999999999},"last":15.648999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":17.195,"5min":16.981999999999999,"15min":22.972999999999999},"min":{"1min":11.882,"5min":9.6539999999999999,"15min":7.4279999999999999},"max":{"1min":22.263999999999999,"5min":22.263999999999999,"15min":235.828},"last":17.268000000000001},{"interface":"front","average":{"1min":16.57,"5min":16.664000000000001,"15min":24.167000000000002},"min":{"1min":11.945,"5min":9.6530000000000005,"15min":9.0869999999999997},"max":{"1min":19.780000000000001,"5min":22.067,"15min":298.74599999999998},"last":18.181999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T00:18:03.831 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-08T00:18:03.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-08T00:18:03.932 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.929+0000 7f17dfd01640 10 monclient: get_monmap_and_config 2023-12-08T00:18:03.932 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.929+0000 7f17dfd01640 10 monclient: build_initial_monmap 2023-12-08T00:18:03.932 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.929+0000 7f17dfd01640 10 monclient: monmap: 2023-12-08T00:18:03.932 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:03.932 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.932 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:03.930987+0000 2023-12-08T00:18:03.932 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:03.930987+0000 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:03.933 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.930+0000 7f17dfd01640 1 Processor -- start 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.930+0000 7f17dfd01640 1 -- start start 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.930+0000 7f17dfd01640 10 monclient: init 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17d806c570 0x7f17d806c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): picked mon.noname-a con 0x7f17d806c570 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80afb60 0x7f17d813f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): picked mon.noname-b con 0x7f17d80afb60 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f17d813f600 0x7f17d81422b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): picked mon.noname-c con 0x7f17d813f600 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d8141e20 con 0x7f17d806c570 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d813fa70 con 0x7f17d813f600 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d806ce90 con 0x7f17d80afb60 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dfd01640 10 monclient(hunting): authenticate will time out at 3284.119279s 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dda9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17d806c570 0x7f17d806c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.931+0000 7f17dd29c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80afb60 0x7f17d813f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f17d813f600 0x7f17d81422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17d806c570 0x7f17d806c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58798/0 (socket says 172.21.15.129:58798) 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 1 -- 172.21.15.129:0/4233011876 learned_addr learned my addr 172.21.15.129:0/4233011876 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:03.937 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 10 monclient(hunting): get_auth_request con 0x7f17d806c570 auth_method 0 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 10 monclient(hunting): get_auth_request con 0x7f17d813f600 auth_method 0 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dd29c640 10 monclient(hunting): get_auth_request con 0x7f17d80afb60 auth_method 0 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dd29c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dd29c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dd29c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dda9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17de29e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dd29c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dd29c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.932+0000 7f17dd29c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 10 monclient(hunting): handle_auth_done global_id 20200 payload 274 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 1 -- 172.21.15.129:0/4233011876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80afb60 msgr2=0x7f17d813f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 1 --2- 172.21.15.129:0/4233011876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80afb60 0x7f17d813f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 1 -- 172.21.15.129:0/4233011876 >> v2:172.21.15.129:3300/0 conn(0x7f17d806c570 msgr2=0x7f17d806c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 1 --2- 172.21.15.129:0/4233011876 >> v2:172.21.15.129:3300/0 conn(0x7f17d806c570 0x7f17d806c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 10 monclient: _finish_hunting 0 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 1 monclient: found mon.noname-c 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.934+0000 7f17de29e640 1 -- 172.21.15.129:0/4233011876 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d80709a0 con 0x7f17d813f600 2023-12-08T00:18:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.936+0000 7f17de29e640 1 --2- 172.21.15.129:0/4233011876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d81422b0 secure :-1 s=READY pgs=4850 cs=0 l=1 rev1=1 crypto rx=0x7f17d806fff0 tx=0x7f17d002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bf4357e2bf8a766 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:03.943 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 1 -- 172.21.15.129:0/4233011876 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002cbd0 con 0x7f17d813f600 2023-12-08T00:18:03.943 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.943 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:03.943 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: dump: 2023-12-08T00:18:03.943 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.943 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: _finish_auth 0 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: _check_auth_tickets 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 1 -- 172.21.15.129:0/4233011876 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d002cd30 con 0x7f17d813f600 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 5 monclient: authenticate success, global_id 20200 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 4 set_mon_vals no callback set 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 1 -- 172.21.15.129:0/4233011876 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002cea0 con 0x7f17d813f600 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dca9b640 10 monclient: dump: 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 10 monclient: shutdown 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 -- 172.21.15.129:0/4233011876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 msgr2=0x7f17d81422b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 --2- 172.21.15.129:0/4233011876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d81422b0 secure :-1 s=READY pgs=4850 cs=0 l=1 rev1=1 crypto rx=0x7f17d806fff0 tx=0x7f17d002be20 comp rx=0 tx=0).stop 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 -- 172.21.15.129:0/4233011876 shutdown_connections 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 --2- 172.21.15.129:0/4233011876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d81422b0 unknown :-1 s=CLOSED pgs=4850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 --2- 172.21.15.129:0/4233011876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80afb60 0x7f17d813f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 --2- 172.21.15.129:0/4233011876 >> v2:172.21.15.129:3300/0 conn(0x7f17d806c570 0x7f17d806c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 -- 172.21.15.129:0/4233011876 >> 172.21.15.129:0/4233011876 conn(0x7f17d8066b90 msgr2=0x7f17d80b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 -- 172.21.15.129:0/4233011876 shutdown_connections 2023-12-08T00:18:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 1 -- 172.21.15.129:0/4233011876 wait complete. 2023-12-08T00:18:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 10 monclient: build_initial_monmap 2023-12-08T00:18:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.941+0000 7f17dfd01640 10 monclient: monmap: 2023-12-08T00:18:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 librados: starting msgr at 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 librados: starting objecter 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 Processor -- start 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 -- start start 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 librados: setting wanted keys 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 librados: calling monclient init 2023-12-08T00:18:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient: init 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d806c570 0x7f17d80ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): picked mon.noname-a con 0x7f17d806c570 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d8132910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): picked mon.noname-b con 0x7f17d813f600 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80ad7d0 0x7f17d8132e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): picked mon.noname-c con 0x7f17d80ad7d0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80aeda0 con 0x7f17d806c570 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80adf60 con 0x7f17d813f600 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80adbf0 con 0x7f17d80ad7d0 2023-12-08T00:18:03.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.942+0000 7f17dfd01640 10 monclient(hunting): authenticate will time out at 3284.130786s 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dda9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d806c570 0x7f17d80ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d8132910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17de29e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80ad7d0 0x7f17d8132e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d8132910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:47926/0 (socket says 172.21.15.129:47926) 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 1 -- 172.21.15.129:0/2477264282 learned_addr learned my addr 172.21.15.129:0/2477264282 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 10 monclient(hunting): get_auth_request con 0x7f17d813f600 auth_method 0 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dda9d640 10 monclient(hunting): get_auth_request con 0x7f17d806c570 auth_method 0 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dda9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dda9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dda9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17de29e640 10 monclient(hunting): get_auth_request con 0x7f17d80ad7d0 auth_method 0 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17de29e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17de29e640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17de29e640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.943+0000 7f17dd29c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.944+0000 7f17dd29c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.944+0000 7f17dda9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.944+0000 7f17dda9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.944+0000 7f17dda9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.944+0000 7f17de29e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.944+0000 7f17de29e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.944+0000 7f17de29e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 10 monclient(hunting): handle_auth_done global_id 20203 payload 824 2023-12-08T00:18:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 1 -- 172.21.15.129:0/2477264282 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80ad7d0 msgr2=0x7f17d8132e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80ad7d0 0x7f17d8132e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 1 -- 172.21.15.129:0/2477264282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d806c570 msgr2=0x7f17d80ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d806c570 0x7f17d80ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 10 monclient: _finish_hunting 0 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 1 monclient: found mon.noname-b 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:03.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.946+0000 7f17dd29c640 1 -- 172.21.15.129:0/2477264282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d0009a70 con 0x7f17d813f600 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.949+0000 7f17dd29c640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d8132910 secure :-1 s=READY pgs=4851 cs=0 l=1 rev1=1 crypto rx=0x7f17c800e9b0 tx=0x7f17c800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.949+0000 7f17ce7fc640 10 client.?.objecter ms_handle_connect 0x7f17d813f600 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.949+0000 7f17ce7fc640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 1 -- 172.21.15.129:0/2477264282 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8009670 con 0x7f17d813f600 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: dump: 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: _finish_auth 0 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: _check_auth_tickets 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 1 -- 172.21.15.129:0/2477264282 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c800c9a0 con 0x7f17d813f600 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 1 -- 172.21.15.129:0/2477264282 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c800cd50 con 0x7f17d813f600 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17dfd01640 5 monclient: authenticate success, global_id 20203 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17dfd01640 10 monclient: _renew_subs 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17dfd01640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d80af4a0 con 0x7f17d813f600 2023-12-08T00:18:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17ce7fc640 10 monclient: dump: 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.952+0000 7f17cffff640 4 set_mon_vals no callback set 2023-12-08T00:18:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 10 client.20203.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 10 monclient: _renew_subs 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d81db090 con 0x7f17d813f600 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 1 librados: init done 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.953+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d806ccb0 con 0x7f17d813f600 2023-12-08T00:18:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.974+0000 7f17ce7fc640 1 -- 172.21.15.129:0/2477264282 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f17c8013470 con 0x7f17d813f600 2023-12-08T00:18:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.975+0000 7f17ce7fc640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f17bc06c820 0x7f17bc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.975+0000 7f17dda9d640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f17bc06c820 0x7f17bc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.975+0000 7f17dda9d640 10 monclient: get_auth_request con 0x7f17bc06c820 auth_method 0 2023-12-08T00:18:03.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.975+0000 7f17ce7fc640 10 client.20203.objecter ms_handle_connect 0x7f17bc06c820 2023-12-08T00:18:03.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.975+0000 7f17dda9d640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f17bc06c820 0x7f17bc06ece0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009bd0 tx=0x7f17d4009240 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:03.980 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.978+0000 7f17ce7fc640 1 -- 172.21.15.129:0/2477264282 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1046..1046 src has 978..1046) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f17c8058d80 con 0x7f17d813f600 2023-12-08T00:18:03.981 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.978+0000 7f17ce7fc640 10 client.20203.objecter ms_dispatch 0x7f17d8000b90 osd_map(1046..1046 src has 978..1046) v4 2023-12-08T00:18:03.981 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.978+0000 7f17ce7fc640 3 client.20203.objecter handle_osd_map got epochs [1046,1046] > 0 2023-12-08T00:18:03.981 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.978+0000 7f17ce7fc640 3 client.20203.objecter handle_osd_map decoding full epoch 1046 2023-12-08T00:18:03.981 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:03.978+0000 7f17ce7fc640 20 client.20203.objecter dump_active .. 0 homeless 2023-12-08T00:18:04.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.211+0000 7f17ce7fc640 1 -- 172.21.15.129:0/2477264282 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17c80550c0 con 0x7f17d813f600 2023-12-08T00:18:04.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.211+0000 7f17ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:04.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.211+0000 7f17ce7fc640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.399+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17d80aef10 con 0x7f17bc06c820 2023-12-08T00:18:04.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.403+0000 7f17ce7fc640 1 -- 172.21.15.129:0/2477264282 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151477 (secure 0 0 0) 0x7f17d80aef10 con 0x7f17bc06c820 2023-12-08T00:18:04.407 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-08T00:18:04.408 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 10 librados: watch_flush enter 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 10 librados: watch_flush exit 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 20 client.20203.objecter shutdown clearing up homeless session... 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 10 client.20203.objecter successfully canceled tick 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f17bc06c820 msgr2=0x7f17bc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f17bc06c820 0x7f17bc06ece0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009bd0 tx=0x7f17d4009240 comp rx=0 tx=0).stop 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 10 monclient: shutdown 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 msgr2=0x7f17d8132910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d8132910 secure :-1 s=READY pgs=4851 cs=0 l=1 rev1=1 crypto rx=0x7f17c800e9b0 tx=0x7f17c800ee70 comp rx=0 tx=0).stop 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 shutdown_connections 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f17bc06c820 0x7f17bc06ece0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80ad7d0 0x7f17d8132e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17d813f600 0x7f17d8132910 unknown :-1 s=CLOSED pgs=4851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 --2- 172.21.15.129:0/2477264282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d806c570 0x7f17d80ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 >> 172.21.15.129:0/2477264282 conn(0x7f17d8066b90 msgr2=0x7f17d80b56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 shutdown_connections 2023-12-08T00:18:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 -- 172.21.15.129:0/2477264282 wait complete. 2023-12-08T00:18:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.406+0000 7f17dfd01640 1 librados: shutdown 2023-12-08T00:18:04.420 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":2011,"stamp":"2023-12-08T00:18:04.136220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":97,"num_write":134,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":594472,"kb_used_data":52984,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754380248,"statfs":{"total":773094113280,"available":772485373952,"internally_reserved":0,"allocated":54255616,"data_stored":26240233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":238,"apply_latency_ms":238,"commit_latency_ns":238000000,"apply_latency_ns":238000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1792,"num_objects":-14,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14,"num_whiteouts":0,"num_read":-12,"num_read_kb":0,"num_write":-14,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.419612"},"pg_stats":[{"pgid":"15.1f","version":"0'0","reported_seq":1001,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.157768+0000","last_change":"2023-12-07T23:47:23.193178+0000","last_active":"2023-12-08T00:18:01.157768+0000","last_peered":"2023-12-08T00:18:01.157768+0000","last_clean":"2023-12-08T00:18:01.157768+0000","last_became_active":"2023-12-07T23:47:23.187382+0000","last_became_peered":"2023-12-07T23:47:23.187382+0000","last_unstale":"2023-12-08T00:18:01.157768+0000","last_undegraded":"2023-12-08T00:18:01.157768+0000","last_fullsized":"2023-12-08T00:18:01.157768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:46:32.619584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":993,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.301223+0000","last_change":"2023-12-07T23:47:26.349171+0000","last_active":"2023-12-08T00:18:01.301223+0000","last_peered":"2023-12-08T00:18:01.301223+0000","last_clean":"2023-12-08T00:18:01.301223+0000","last_became_active":"2023-12-07T23:47:26.088721+0000","last_became_peered":"2023-12-07T23:47:26.088721+0000","last_unstale":"2023-12-08T00:18:01.301223+0000","last_undegraded":"2023-12-08T00:18:01.301223+0000","last_fullsized":"2023-12-08T00:18:01.301223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:46:10.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1010,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.029374+0000","last_change":"2023-12-07T23:47:18.674241+0000","last_active":"2023-12-08T00:17:45.029374+0000","last_peered":"2023-12-08T00:17:45.029374+0000","last_clean":"2023-12-08T00:17:45.029374+0000","last_became_active":"2023-12-07T23:47:18.658074+0000","last_became_peered":"2023-12-07T23:47:18.658074+0000","last_unstale":"2023-12-08T00:17:45.029374+0000","last_undegraded":"2023-12-08T00:17:45.029374+0000","last_fullsized":"2023-12-08T00:17:45.029374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T00:31:54.317455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":987,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.798505+0000","last_change":"2023-12-07T23:47:39.142806+0000","last_active":"2023-12-08T00:18:00.798505+0000","last_peered":"2023-12-08T00:18:00.798505+0000","last_clean":"2023-12-08T00:18:00.798505+0000","last_became_active":"2023-12-07T23:47:39.133797+0000","last_became_peered":"2023-12-07T23:47:39.133797+0000","last_unstale":"2023-12-08T00:18:00.798505+0000","last_undegraded":"2023-12-08T00:18:00.798505+0000","last_fullsized":"2023-12-08T00:18:00.798505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:38:14.932982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":991,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.056223+0000","last_change":"2023-12-07T23:47:20.151369+0000","last_active":"2023-12-08T00:17:45.056223+0000","last_peered":"2023-12-08T00:17:45.056223+0000","last_clean":"2023-12-08T00:17:45.056223+0000","last_became_active":"2023-12-07T23:47:20.132690+0000","last_became_peered":"2023-12-07T23:47:20.132690+0000","last_unstale":"2023-12-08T00:17:45.056223+0000","last_undegraded":"2023-12-08T00:17:45.056223+0000","last_fullsized":"2023-12-08T00:17:45.056223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:51:11.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":992,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.267328+0000","last_change":"2023-12-07T23:47:19.053664+0000","last_active":"2023-12-08T00:18:01.267328+0000","last_peered":"2023-12-08T00:18:01.267328+0000","last_clean":"2023-12-08T00:18:01.267328+0000","last_became_active":"2023-12-07T23:47:18.926939+0000","last_became_peered":"2023-12-07T23:47:18.926939+0000","last_unstale":"2023-12-08T00:18:01.267328+0000","last_undegraded":"2023-12-08T00:18:01.267328+0000","last_fullsized":"2023-12-08T00:18:01.267328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:28:02.327836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1","version":"64'7","reported_seq":1006,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.725038+0000","last_change":"2023-12-07T23:47:15.440534+0000","last_active":"2023-12-08T00:18:00.725038+0000","last_peered":"2023-12-08T00:18:00.725038+0000","last_clean":"2023-12-08T00:18:00.725038+0000","last_became_active":"2023-12-07T23:47:15.386185+0000","last_became_peered":"2023-12-07T23:47:15.386185+0000","last_unstale":"2023-12-08T00:18:00.725038+0000","last_undegraded":"2023-12-08T00:18:00.725038+0000","last_fullsized":"2023-12-08T00:18:00.725038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:40:51.648331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":993,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.242191+0000","last_change":"2023-12-07T23:47:18.719911+0000","last_active":"2023-12-08T00:18:01.242191+0000","last_peered":"2023-12-08T00:18:01.242191+0000","last_clean":"2023-12-08T00:18:01.242191+0000","last_became_active":"2023-12-07T23:47:18.700591+0000","last_became_peered":"2023-12-07T23:47:18.700591+0000","last_unstale":"2023-12-08T00:18:01.242191+0000","last_undegraded":"2023-12-08T00:18:01.242191+0000","last_fullsized":"2023-12-08T00:18:01.242191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:15:06.752653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":992,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.308419+0000","last_change":"2023-12-07T23:47:16.106412+0000","last_active":"2023-12-08T00:18:01.308419+0000","last_peered":"2023-12-08T00:18:01.308419+0000","last_clean":"2023-12-08T00:18:01.308419+0000","last_became_active":"2023-12-07T23:47:16.101044+0000","last_became_peered":"2023-12-07T23:47:16.101044+0000","last_unstale":"2023-12-08T00:18:01.308419+0000","last_undegraded":"2023-12-08T00:18:01.308419+0000","last_fullsized":"2023-12-08T00:18:01.308419+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:51:45.972358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":992,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.299033+0000","last_change":"2023-12-07T23:47:19.725523+0000","last_active":"2023-12-08T00:18:01.299033+0000","last_peered":"2023-12-08T00:18:01.299033+0000","last_clean":"2023-12-08T00:18:01.299033+0000","last_became_active":"2023-12-07T23:47:18.123992+0000","last_became_peered":"2023-12-07T23:47:18.123992+0000","last_unstale":"2023-12-08T00:18:01.299033+0000","last_undegraded":"2023-12-08T00:18:01.299033+0000","last_fullsized":"2023-12-08T00:18:01.299033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:05:44.580089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":991,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.071367+0000","last_change":"2023-12-07T23:47:19.649219+0000","last_active":"2023-12-08T00:17:45.071367+0000","last_peered":"2023-12-08T00:17:45.071367+0000","last_clean":"2023-12-08T00:17:45.071367+0000","last_became_active":"2023-12-07T23:47:19.507339+0000","last_became_peered":"2023-12-07T23:47:19.507339+0000","last_unstale":"2023-12-08T00:17:45.071367+0000","last_undegraded":"2023-12-08T00:17:45.071367+0000","last_fullsized":"2023-12-08T00:17:45.071367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:38:46.882596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1010,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.246275+0000","last_change":"2023-12-07T23:48:06.632530+0000","last_active":"2023-12-08T00:18:01.246275+0000","last_peered":"2023-12-08T00:18:01.246275+0000","last_clean":"2023-12-08T00:18:01.246275+0000","last_became_active":"2023-12-07T23:48:06.627236+0000","last_became_peered":"2023-12-07T23:48:06.627236+0000","last_unstale":"2023-12-08T00:18:01.246275+0000","last_undegraded":"2023-12-08T00:18:01.246275+0000","last_fullsized":"2023-12-08T00:18:01.246275+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:43:12.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1009,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.274984+0000","last_change":"2023-12-07T23:47:18.049591+0000","last_active":"2023-12-08T00:18:01.274984+0000","last_peered":"2023-12-08T00:18:01.274984+0000","last_clean":"2023-12-08T00:18:01.274984+0000","last_became_active":"2023-12-07T23:47:18.028936+0000","last_became_peered":"2023-12-07T23:47:18.028936+0000","last_unstale":"2023-12-08T00:18:01.274984+0000","last_undegraded":"2023-12-08T00:18:01.274984+0000","last_fullsized":"2023-12-08T00:18:01.274984+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:11:43.305002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"841'88","reported_seq":1119,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.733737+0000","last_change":"2023-12-07T23:46:05.583767+0000","last_active":"2023-12-08T00:18:00.733737+0000","last_peered":"2023-12-08T00:18:00.733737+0000","last_clean":"2023-12-08T00:18:00.733737+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-08T00:18:00.733737+0000","last_undegraded":"2023-12-08T00:18:00.733737+0000","last_fullsized":"2023-12-08T00:18:00.733737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:02.581722+0000","last_clean_scrub_stamp":"2023-12-07T23:46:02.581722+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:48:03.670610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.057811+0000","last_change":"2023-12-07T23:46:13.815701+0000","last_active":"2023-12-08T00:17:45.057811+0000","last_peered":"2023-12-08T00:17:45.057811+0000","last_clean":"2023-12-08T00:17:45.057811+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-08T00:17:45.057811+0000","last_undegraded":"2023-12-08T00:17:45.057811+0000","last_fullsized":"2023-12-08T00:17:45.057811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:13.731531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":992,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.756380+0000","last_change":"2023-12-07T23:47:18.113515+0000","last_active":"2023-12-08T00:18:00.756380+0000","last_peered":"2023-12-08T00:18:00.756380+0000","last_clean":"2023-12-08T00:18:00.756380+0000","last_became_active":"2023-12-07T23:47:18.104193+0000","last_became_peered":"2023-12-07T23:47:18.104193+0000","last_unstale":"2023-12-08T00:18:00.756380+0000","last_undegraded":"2023-12-08T00:18:00.756380+0000","last_fullsized":"2023-12-08T00:18:00.756380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:39:47.005386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.013885+0000","last_change":"2023-12-07T23:46:13.418137+0000","last_active":"2023-12-08T00:17:45.013885+0000","last_peered":"2023-12-08T00:17:45.013885+0000","last_clean":"2023-12-08T00:17:45.013885+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-08T00:17:45.013885+0000","last_undegraded":"2023-12-08T00:17:45.013885+0000","last_fullsized":"2023-12-08T00:17:45.013885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:51:13.255892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1006,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.741729+0000","last_change":"2023-12-07T23:47:25.704244+0000","last_active":"2023-12-08T00:18:00.741729+0000","last_peered":"2023-12-08T00:18:00.741729+0000","last_clean":"2023-12-08T00:18:00.741729+0000","last_became_active":"2023-12-07T23:47:25.699009+0000","last_became_peered":"2023-12-07T23:47:25.699009+0000","last_unstale":"2023-12-08T00:18:00.741729+0000","last_undegraded":"2023-12-08T00:18:00.741729+0000","last_fullsized":"2023-12-08T00:18:00.741729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:23:03.360563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1005,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.112899+0000","last_change":"2023-12-07T23:46:13.343801+0000","last_active":"2023-12-08T00:18:01.112899+0000","last_peered":"2023-12-08T00:18:01.112899+0000","last_clean":"2023-12-08T00:18:01.112899+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-08T00:18:01.112899+0000","last_undegraded":"2023-12-08T00:18:01.112899+0000","last_fullsized":"2023-12-08T00:18:01.112899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:42:20.201085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1005,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.762618+0000","last_change":"2023-12-07T23:47:26.181304+0000","last_active":"2023-12-08T00:18:00.762618+0000","last_peered":"2023-12-08T00:18:00.762618+0000","last_clean":"2023-12-08T00:18:00.762618+0000","last_became_active":"2023-12-07T23:47:26.173821+0000","last_became_peered":"2023-12-07T23:47:26.173821+0000","last_unstale":"2023-12-08T00:18:00.762618+0000","last_undegraded":"2023-12-08T00:18:00.762618+0000","last_fullsized":"2023-12-08T00:18:00.762618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T23:47:56.552622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.028661+0000","last_change":"2023-12-07T23:46:13.873757+0000","last_active":"2023-12-08T00:17:45.028661+0000","last_peered":"2023-12-08T00:17:45.028661+0000","last_clean":"2023-12-08T00:17:45.028661+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-08T00:17:45.028661+0000","last_undegraded":"2023-12-08T00:17:45.028661+0000","last_fullsized":"2023-12-08T00:17:45.028661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:40:06.824492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1007,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.727055+0000","last_change":"2023-12-07T23:47:46.300836+0000","last_active":"2023-12-08T00:18:00.727055+0000","last_peered":"2023-12-08T00:18:00.727055+0000","last_clean":"2023-12-08T00:18:00.727055+0000","last_became_active":"2023-12-07T23:47:46.295177+0000","last_became_peered":"2023-12-07T23:47:46.295177+0000","last_unstale":"2023-12-08T00:18:00.727055+0000","last_undegraded":"2023-12-08T00:18:00.727055+0000","last_fullsized":"2023-12-08T00:18:00.727055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:54:00.508066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":997,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.287703+0000","last_change":"2023-12-07T23:46:13.905749+0000","last_active":"2023-12-08T00:18:01.287703+0000","last_peered":"2023-12-08T00:18:01.287703+0000","last_clean":"2023-12-08T00:18:01.287703+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-08T00:18:01.287703+0000","last_undegraded":"2023-12-08T00:18:01.287703+0000","last_fullsized":"2023-12-08T00:18:01.287703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:45:13.859049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":989,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.747705+0000","last_change":"2023-12-07T23:47:18.243486+0000","last_active":"2023-12-08T00:18:00.747705+0000","last_peered":"2023-12-08T00:18:00.747705+0000","last_clean":"2023-12-08T00:18:00.747705+0000","last_became_active":"2023-12-07T23:47:18.206551+0000","last_became_peered":"2023-12-07T23:47:18.206551+0000","last_unstale":"2023-12-08T00:18:00.747705+0000","last_undegraded":"2023-12-08T00:18:00.747705+0000","last_fullsized":"2023-12-08T00:18:00.747705+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:33:27.429730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":996,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.312242+0000","last_change":"2023-12-07T23:46:13.447293+0000","last_active":"2023-12-08T00:17:45.312242+0000","last_peered":"2023-12-08T00:17:45.312242+0000","last_clean":"2023-12-08T00:17:45.312242+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-08T00:17:45.312242+0000","last_undegraded":"2023-12-08T00:17:45.312242+0000","last_fullsized":"2023-12-08T00:17:45.312242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:00.005960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":991,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.323818+0000","last_change":"2023-12-07T23:47:15.786542+0000","last_active":"2023-12-08T00:18:01.323818+0000","last_peered":"2023-12-08T00:18:01.323818+0000","last_clean":"2023-12-08T00:18:01.323818+0000","last_became_active":"2023-12-07T23:47:15.770063+0000","last_became_peered":"2023-12-07T23:47:15.770063+0000","last_unstale":"2023-12-08T00:18:01.323818+0000","last_undegraded":"2023-12-08T00:18:01.323818+0000","last_fullsized":"2023-12-08T00:18:01.323818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:18:15.429980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1014,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.277123+0000","last_change":"2023-12-07T23:46:13.656681+0000","last_active":"2023-12-08T00:18:01.277123+0000","last_peered":"2023-12-08T00:18:01.277123+0000","last_clean":"2023-12-08T00:18:01.277123+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-08T00:18:01.277123+0000","last_undegraded":"2023-12-08T00:18:01.277123+0000","last_fullsized":"2023-12-08T00:18:01.277123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:29.359895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":992,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.265153+0000","last_change":"2023-12-07T23:47:15.701780+0000","last_active":"2023-12-08T00:18:01.265153+0000","last_peered":"2023-12-08T00:18:01.265153+0000","last_clean":"2023-12-08T00:18:01.265153+0000","last_became_active":"2023-12-07T23:47:15.693158+0000","last_became_peered":"2023-12-07T23:47:15.693158+0000","last_unstale":"2023-12-08T00:18:01.265153+0000","last_undegraded":"2023-12-08T00:18:01.265153+0000","last_fullsized":"2023-12-08T00:18:01.265153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:12.790526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":995,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.078677+0000","last_change":"2023-12-07T23:46:13.620414+0000","last_active":"2023-12-08T00:17:45.078677+0000","last_peered":"2023-12-08T00:17:45.078677+0000","last_clean":"2023-12-08T00:17:45.078677+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-08T00:17:45.078677+0000","last_undegraded":"2023-12-08T00:17:45.078677+0000","last_fullsized":"2023-12-08T00:17:45.078677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:44:19.783064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":999,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.122031+0000","last_change":"2023-12-07T23:47:21.241228+0000","last_active":"2023-12-08T00:18:01.122031+0000","last_peered":"2023-12-08T00:18:01.122031+0000","last_clean":"2023-12-08T00:18:01.122031+0000","last_became_active":"2023-12-07T23:47:21.229799+0000","last_became_peered":"2023-12-07T23:47:21.229799+0000","last_unstale":"2023-12-08T00:18:01.122031+0000","last_undegraded":"2023-12-08T00:18:01.122031+0000","last_fullsized":"2023-12-08T00:18:01.122031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:30:12.922388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":991,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.722464+0000","last_change":"2023-12-07T23:47:36.622692+0000","last_active":"2023-12-08T00:18:00.722464+0000","last_peered":"2023-12-08T00:18:00.722464+0000","last_clean":"2023-12-08T00:18:00.722464+0000","last_became_active":"2023-12-07T23:47:36.616695+0000","last_became_peered":"2023-12-07T23:47:36.616695+0000","last_unstale":"2023-12-08T00:18:00.722464+0000","last_undegraded":"2023-12-08T00:18:00.722464+0000","last_fullsized":"2023-12-08T00:18:00.722464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:48:24.932348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":991,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.731548+0000","last_change":"2023-12-07T23:47:36.574414+0000","last_active":"2023-12-08T00:18:00.731548+0000","last_peered":"2023-12-08T00:18:00.731548+0000","last_clean":"2023-12-08T00:18:00.731548+0000","last_became_active":"2023-12-07T23:47:36.567130+0000","last_became_peered":"2023-12-07T23:47:36.567130+0000","last_unstale":"2023-12-08T00:18:00.731548+0000","last_undegraded":"2023-12-08T00:18:00.731548+0000","last_fullsized":"2023-12-08T00:18:00.731548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:53:21.050211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":989,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.054680+0000","last_change":"2023-12-07T23:47:37.286320+0000","last_active":"2023-12-08T00:17:45.054680+0000","last_peered":"2023-12-08T00:17:45.054680+0000","last_clean":"2023-12-08T00:17:45.054680+0000","last_became_active":"2023-12-07T23:47:37.279080+0000","last_became_peered":"2023-12-07T23:47:37.279080+0000","last_unstale":"2023-12-08T00:17:45.054680+0000","last_undegraded":"2023-12-08T00:17:45.054680+0000","last_fullsized":"2023-12-08T00:17:45.054680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:43:22.378569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":990,"reported_epoch":1042,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.866514+0000","last_change":"2023-12-07T23:47:21.394853+0000","last_active":"2023-12-08T00:17:45.866514+0000","last_peered":"2023-12-08T00:17:45.866514+0000","last_clean":"2023-12-08T00:17:45.866514+0000","last_became_active":"2023-12-07T23:47:21.389382+0000","last_became_peered":"2023-12-07T23:47:21.389382+0000","last_unstale":"2023-12-08T00:17:45.866514+0000","last_undegraded":"2023-12-08T00:17:45.866514+0000","last_fullsized":"2023-12-08T00:17:45.866514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:12:18.044930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1005,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.751316+0000","last_change":"2023-12-07T23:47:20.975428+0000","last_active":"2023-12-08T00:18:00.751316+0000","last_peered":"2023-12-08T00:18:00.751316+0000","last_clean":"2023-12-08T00:18:00.751316+0000","last_became_active":"2023-12-07T23:47:20.968485+0000","last_became_peered":"2023-12-07T23:47:20.968485+0000","last_unstale":"2023-12-08T00:18:00.751316+0000","last_undegraded":"2023-12-08T00:18:00.751316+0000","last_fullsized":"2023-12-08T00:18:00.751316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:03:02.615974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":987,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.786637+0000","last_change":"2023-12-07T23:47:21.404597+0000","last_active":"2023-12-08T00:18:00.786637+0000","last_peered":"2023-12-08T00:18:00.786637+0000","last_clean":"2023-12-08T00:18:00.786637+0000","last_became_active":"2023-12-07T23:47:21.395304+0000","last_became_peered":"2023-12-07T23:47:21.395304+0000","last_unstale":"2023-12-08T00:18:00.786637+0000","last_undegraded":"2023-12-08T00:18:00.786637+0000","last_fullsized":"2023-12-08T00:18:00.786637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:02:02.413893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1004,"reported_epoch":1044,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.759930+0000","last_change":"2023-12-07T23:47:38.795823+0000","last_active":"2023-12-08T00:18:00.759930+0000","last_peered":"2023-12-08T00:18:00.759930+0000","last_clean":"2023-12-08T00:18:00.759930+0000","last_became_active":"2023-12-07T23:47:38.790323+0000","last_became_peered":"2023-12-07T23:47:38.790323+0000","last_unstale":"2023-12-08T00:18:00.759930+0000","last_undegraded":"2023-12-08T00:18:00.759930+0000","last_fullsized":"2023-12-08T00:18:00.759930+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:40:01.316349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":990,"reported_epoch":1041,"state":"active+clean","last_fresh":"2023-12-08T00:17:45.015499+0000","last_change":"2023-12-07T23:47:37.187628+0000","last_active":"2023-12-08T00:17:45.015499+0000","last_peered":"2023-12-08T00:17:45.015499+0000","last_clean":"2023-12-08T00:17:45.015499+0000","last_became_active":"2023-12-07T23:47:37.180489+0000","last_became_peered":"2023-12-07T23:47:37.180489+0000","last_unstale":"2023-12-08T00:17:45.015499+0000","last_undegraded":"2023-12-08T00:17:45.015499+0000","last_fullsized":"2023-12-08T00:17:45.015499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:28:21.180560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1009,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.219768+0000","last_change":"2023-12-07T23:48:09.211978+0000","last_active":"2023-12-08T00:18:01.219768+0000","last_peered":"2023-12-08T00:18:01.219768+0000","last_clean":"2023-12-08T00:18:01.219768+0000","last_became_active":"2023-12-07T23:48:09.186728+0000","last_became_peered":"2023-12-07T23:48:09.186728+0000","last_unstale":"2023-12-08T00:18:01.219768+0000","last_undegraded":"2023-12-08T00:18:01.219768+0000","last_fullsized":"2023-12-08T00:18:01.219768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:09:04.978015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":988,"reported_epoch":1043,"state":"active+clean","last_fresh":"2023-12-08T00:18:00.764294+0000","last_change":"2023-12-07T23:47:20.131095+0000","last_active":"2023-12-08T00:18:00.764294+0000","last_peered":"2023-12-08T00:18:00.764294+0000","last_clean":"2023-12-08T00:18:00.764294+0000","last_became_active":"2023-12-07T23:47:20.123312+0000","last_became_peered":"2023-12-07T23:47:20.123312+0000","last_unstale":"2023-12-08T00:18:00.764294+0000","last_undegraded":"2023-12-08T00:18:00.764294+0000","last_fullsized":"2023-12-08T00:18:00.764294+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:27:53.246959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1009,"reported_epoch":1045,"state":"active+clean","last_fresh":"2023-12-08T00:18:01.253320+0000","last_change":"2023-12-07T23:47:18.960597+0000","last_active":"2023-12-08T00:18:01.253320+0000","last_peered":"2023-12-08T00:18:01.253320+0000","last_clean":"2023-12-08T00:18:01.253320+0000","last_became_active":"2023-12-07T23:47:18.946564+0000","last_became_peered":"2023-12-07T23:47:18.946564+0000","last_unstale":"2023-12-08T00:18:01.253320+0000","last_undegraded":"2023-12-08T00:18:01.253320+0000","last_fullsized":"2023-12-08T00:18:01.253320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:55:18.226923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":2,"acting":2,"num_store_stats":2},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477119,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82556,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289284,"statfs":{"total":96636764160,"available":96552226816,"internally_reserved":0,"allocated":7131136,"data_stored":3426749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":8.3040000000000003,"5min":8.2520000000000007,"15min":15.449999999999999},"min":{"1min":4.4939999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"max":{"1min":20.571999999999999,"5min":20.571999999999999,"15min":272.77100000000002},"last":17.207999999999998},{"interface":"front","average":{"1min":11.718,"5min":10.836,"15min":18.363},"min":{"1min":5.5990000000000002,"5min":4.9900000000000002,"15min":4.9649999999999999},"max":{"1min":47.829999999999998,"5min":47.829999999999998,"15min":288.42000000000002},"last":17.963000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.33,"5min":11.922000000000001,"15min":21.003},"min":{"1min":6.2279999999999998,"5min":6.2279999999999998,"15min":5.7649999999999997},"max":{"1min":73.218999999999994,"5min":73.218999999999994,"15min":320.57900000000001},"last":10.895},{"interface":"front","average":{"1min":16.672000000000001,"5min":13.327,"15min":23.423999999999999},"min":{"1min":9.1400000000000006,"5min":6.7279999999999998,"15min":5.5670000000000002},"max":{"1min":91.882000000000005,"5min":91.882000000000005,"15min":461.51799999999997},"last":20.463000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:38 2023","interfaces":[{"interface":"back","average":{"1min":11.853,"5min":9.7810000000000006,"15min":15.744},"min":{"1min":7.343,"5min":6.0389999999999997,"15min":5.6959999999999997},"max":{"1min":66.941999999999993,"5min":66.941999999999993,"15min":246.47499999999999},"last":19.763999999999999},{"interface":"front","average":{"1min":17.085000000000001,"5min":14.343,"15min":22.997},"min":{"1min":8.7669999999999995,"5min":8.7279999999999998,"15min":7.3079999999999998},"max":{"1min":69.078999999999994,"5min":69.078999999999994,"15min":356.02100000000002},"last":21.995999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":17.209,"5min":14.24,"15min":19.495999999999999},"min":{"1min":7.7300000000000004,"5min":7.048,"15min":6.7690000000000001},"max":{"1min":107.04900000000001,"5min":107.04900000000001,"15min":180.76400000000001},"last":14.782999999999999},{"interface":"front","average":{"1min":16.361000000000001,"5min":14.583,"15min":22.693000000000001},"min":{"1min":10.125,"5min":6.5629999999999997,"15min":6.2990000000000004},"max":{"1min":62.020000000000003,"5min":62.020000000000003,"15min":223.91300000000001},"last":21.356999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.823,"5min":15.294,"15min":24.542000000000002},"min":{"1min":10.445,"5min":8.9920000000000009,"15min":7.2889999999999997},"max":{"1min":19.890000000000001,"5min":29.309999999999999,"15min":251.892},"last":18.597999999999999},{"interface":"front","average":{"1min":29.396999999999998,"5min":19.707000000000001,"15min":28.649999999999999},"min":{"1min":13.223000000000001,"5min":11.711,"15min":10.568},"max":{"1min":265.69799999999998,"5min":265.69799999999998,"15min":279.791},"last":23.388000000000002}]},{"osd":6,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":21.244,"5min":14.326000000000001,"15min":19.547000000000001},"min":{"1min":7.2850000000000001,"5min":7.2850000000000001,"15min":7.2850000000000001},"max":{"1min":214.774,"5min":214.774,"15min":214.774},"last":12.128},{"interface":"front","average":{"1min":26.649999999999999,"5min":17.835000000000001,"15min":24.134},"min":{"1min":12.226000000000001,"5min":10.428000000000001,"15min":8.1440000000000001},"max":{"1min":229.54300000000001,"5min":229.54300000000001,"15min":229.54300000000001},"last":13.499000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":16.917000000000002,"5min":17.433,"15min":28.428000000000001},"min":{"1min":8.923,"5min":8.923,"15min":8.923},"max":{"1min":21.625,"5min":27.763999999999999,"15min":281.31},"last":24.106000000000002},{"interface":"front","average":{"1min":14.558999999999999,"5min":14.391999999999999,"15min":23.718},"min":{"1min":9.9049999999999994,"5min":8.5600000000000005,"15min":7.649},"max":{"1min":23.234999999999999,"5min":23.234999999999999,"15min":255.72499999999999},"last":15.93}]}]},{"osd":2,"up_from":16,"seq":68719477120,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69880,"kb_used_data":6292,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301960,"statfs":{"total":96636764160,"available":96565207040,"internally_reserved":0,"allocated":6443008,"data_stored":3053431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5220000000000002,"5min":8.8870000000000005,"15min":15.034000000000001},"min":{"1min":4.8140000000000001,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":20.408000000000001,"5min":20.408000000000001,"15min":249.42699999999999},"last":15.491},{"interface":"front","average":{"1min":9.9410000000000007,"5min":9.4969999999999999,"15min":15.816000000000001},"min":{"1min":5.9480000000000004,"5min":5.8230000000000004,"15min":5.1310000000000002},"max":{"1min":19.068999999999999,"5min":19.068999999999999,"15min":241.45599999999999},"last":11.285}]},{"osd":1,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":12.634,"5min":11.613,"15min":19.722000000000001},"min":{"1min":8.1159999999999997,"5min":6.8179999999999996,"15min":6.766},"max":{"1min":19.122,"5min":19.122,"15min":352.21100000000001},"last":12.108000000000001},{"interface":"front","average":{"1min":11.063000000000001,"5min":10.92,"15min":17.779},"min":{"1min":8.3200000000000003,"5min":6.2720000000000002,"15min":5.9109999999999996},"max":{"1min":14.936,"5min":18.652000000000001,"15min":327.81099999999998},"last":13.92}]},{"osd":3,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.925000000000001,"5min":10.75,"15min":19.754000000000001},"min":{"1min":7.798,"5min":6.2770000000000001,"15min":6.1550000000000002},"max":{"1min":15.944000000000001,"5min":19.123999999999999,"15min":287.56299999999999},"last":8.1010000000000009},{"interface":"front","average":{"1min":14.613,"5min":13.928000000000001,"15min":21.443999999999999},"min":{"1min":8.9900000000000002,"5min":8.3059999999999992,"15min":7.149},"max":{"1min":17.920000000000002,"5min":20.739000000000001,"15min":255.334},"last":16.969000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":15.124000000000001,"5min":15.16,"15min":23.259},"min":{"1min":8.9789999999999992,"5min":8.9220000000000006,"15min":8.1539999999999999},"max":{"1min":19.739999999999998,"5min":20.571000000000002,"15min":301.12799999999999},"last":20.748999999999999},{"interface":"front","average":{"1min":12.436999999999999,"5min":12.302,"15min":20.571000000000002},"min":{"1min":6.7450000000000001,"5min":6.7450000000000001,"15min":6.2649999999999997},"max":{"1min":19.678999999999998,"5min":23.859999999999999,"15min":348.38099999999997},"last":10.226000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":16.494,"15min":24.294},"min":{"1min":14.236000000000001,"5min":9.0410000000000004,"15min":8.5470000000000006},"max":{"1min":20.163,"5min":22.094000000000001,"15min":289.18700000000001},"last":23.734000000000002},{"interface":"front","average":{"1min":14.500999999999999,"5min":14.137,"15min":20.013000000000002},"min":{"1min":9.3870000000000005,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":20.331,"5min":20.667000000000002,"15min":248.63499999999999},"last":21.385000000000002}]},{"osd":6,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":16.123000000000001,"5min":15.634,"15min":21.067},"min":{"1min":10.611000000000001,"5min":9.9610000000000003,"15min":7.4240000000000004},"max":{"1min":19.946000000000002,"5min":21.585000000000001,"15min":349.41399999999999},"last":41.552999999999997},{"interface":"front","average":{"1min":11.491,"5min":12.48,"15min":20.643999999999998},"min":{"1min":7.431,"5min":7.423,"15min":7.423},"max":{"1min":19.212,"5min":21.007000000000001,"15min":231.422},"last":34.932000000000002}]},{"osd":7,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":13.156000000000001,"5min":14.472,"15min":24.436},"min":{"1min":8.6010000000000009,"5min":8.6010000000000009,"15min":8.6010000000000009},"max":{"1min":20.013999999999999,"5min":20.428000000000001,"15min":350.52199999999999},"last":16.106999999999999},{"interface":"front","average":{"1min":16.969000000000001,"5min":17.178000000000001,"15min":24.536000000000001},"min":{"1min":11.411,"5min":10.608000000000001,"15min":10.475},"max":{"1min":20.597000000000001,"5min":22.510999999999999,"15min":235.19200000000001},"last":14.58}]}]},{"osd":0,"up_from":16,"seq":68719477119,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70076,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301764,"statfs":{"total":96636764160,"available":96565006336,"internally_reserved":0,"allocated":7131136,"data_stored":3426749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 8 00:17:29 2023","interfaces":[{"interface":"back","average":{"1min":10.356,"5min":9.7119999999999997,"15min":18.596},"min":{"1min":5.2789999999999999,"5min":4.827,"15min":4.5830000000000002},"max":{"1min":38.834000000000003,"5min":38.834000000000003,"15min":317.99599999999998},"last":14.888999999999999},{"interface":"front","average":{"1min":11.698,"5min":9.2210000000000001,"15min":17.318999999999999},"min":{"1min":6.6840000000000002,"5min":4.9139999999999997,"15min":4.9139999999999997},"max":{"1min":49.262,"5min":49.262,"15min":313.14400000000001},"last":7.2000000000000002}]},{"osd":2,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":30.468,"5min":16.053000000000001,"15min":20.640000000000001},"min":{"1min":8.9280000000000008,"5min":7.1600000000000001,"15min":6.4740000000000002},"max":{"1min":196.96700000000001,"5min":196.96700000000001,"15min":228.02500000000001},"last":11.413},{"interface":"front","average":{"1min":23.956,"5min":14.147,"15min":18.375},"min":{"1min":6.8819999999999997,"5min":5.7530000000000001,"15min":5.6059999999999999},"max":{"1min":168.91300000000001,"5min":168.91300000000001,"15min":197.78899999999999},"last":8.3989999999999991}]},{"osd":3,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":14.613,"5min":13.257,"15min":22.777999999999999},"min":{"1min":10.614000000000001,"5min":8.0500000000000007,"15min":7.0069999999999997},"max":{"1min":19.207999999999998,"5min":19.207999999999998,"15min":239.50999999999999},"last":12.526},{"interface":"front","average":{"1min":16.495999999999999,"5min":16.079000000000001,"15min":24.495999999999999},"min":{"1min":10.835000000000001,"5min":10.667,"15min":9.141},"max":{"1min":20.774000000000001,"5min":29.132999999999999,"15min":241.839},"last":15.701000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":8.8249999999999993,"5min":8.9369999999999994,"15min":14.958},"min":{"1min":6.4029999999999996,"5min":5.5570000000000004,"15min":5.5430000000000001},"max":{"1min":22.925000000000001,"5min":22.925000000000001,"15min":211.452},"last":10.794},{"interface":"front","average":{"1min":16.763999999999999,"5min":16.686,"15min":22.658000000000001},"min":{"1min":10.212999999999999,"5min":10.212999999999999,"15min":9.3030000000000008},"max":{"1min":24.173999999999999,"5min":29.969000000000001,"15min":212.459},"last":16.332000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":21.459,"5min":14.852,"15min":19.033999999999999},"min":{"1min":8.6750000000000007,"5min":7.8970000000000002,"15min":5.9740000000000002},"max":{"1min":175.68100000000001,"5min":175.68100000000001,"15min":175.68100000000001},"last":9.8079999999999998},{"interface":"front","average":{"1min":22.960000000000001,"5min":13.959,"15min":19.902000000000001},"min":{"1min":8.1950000000000003,"5min":6.7789999999999999,"15min":6.6660000000000004},"max":{"1min":186.625,"5min":186.625,"15min":186.625},"last":13.194000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":17.789999999999999,"15min":26.085000000000001},"min":{"1min":11.406000000000001,"5min":11.406000000000001,"15min":10.641999999999999},"max":{"1min":26.757999999999999,"5min":30.727,"15min":227.892},"last":17.099},{"interface":"front","average":{"1min":16.129999999999999,"5min":14.592000000000001,"15min":20.123999999999999},"min":{"1min":10.122,"5min":8.2919999999999998,"15min":8.2919999999999998},"max":{"1min":44.432000000000002,"5min":44.432000000000002,"15min":153.59700000000001},"last":14.284000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.475000000000001,"5min":16.189,"15min":24.940000000000001},"min":{"1min":11.638,"5min":9.3550000000000004,"15min":8.9369999999999994},"max":{"1min":39.835999999999999,"5min":39.835999999999999,"15min":264.44600000000003},"last":19.535},{"interface":"front","average":{"1min":14.044,"5min":12.993,"15min":20.210000000000001},"min":{"1min":7.8209999999999997,"5min":7.5350000000000001,"15min":7.5},"max":{"1min":41.091000000000001,"5min":41.091000000000001,"15min":243.84899999999999},"last":17.846}]}]},{"osd":1,"up_from":14,"seq":60129542525,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70520,"kb_used_data":6936,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301320,"statfs":{"total":96636764160,"available":96564551680,"internally_reserved":0,"allocated":7102464,"data_stored":3577342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.6829999999999998,"5min":9.3740000000000006,"15min":17.443000000000001},"min":{"1min":5.8540000000000001,"5min":5.6449999999999996,"15min":4.2539999999999996},"max":{"1min":17.231000000000002,"5min":18.986999999999998,"15min":324.46600000000001},"last":7.7610000000000001},{"interface":"front","average":{"1min":14.244,"5min":13.406000000000001,"15min":22.675000000000001},"min":{"1min":7.2469999999999999,"5min":6.4950000000000001,"15min":5.5369999999999999},"max":{"1min":21.832000000000001,"5min":21.832000000000001,"15min":342.39299999999997},"last":10.605}]},{"osd":2,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.9380000000000006,"5min":9.7550000000000008,"15min":20.015000000000001},"min":{"1min":5.6289999999999996,"5min":4.7409999999999997,"15min":4.7409999999999997},"max":{"1min":15.329000000000001,"5min":18.332000000000001,"15min":309.75999999999999},"last":83.418999999999997},{"interface":"front","average":{"1min":13.932,"5min":13.237,"15min":23.968},"min":{"1min":7.7919999999999998,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":19.77,"5min":22.640999999999998,"15min":343.49900000000002},"last":17.074999999999999}]},{"osd":3,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":13.263,"5min":14.33,"15min":22.765000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"max":{"1min":17.984999999999999,"5min":20.395,"15min":345.07600000000002},"last":11.33},{"interface":"front","average":{"1min":10.297000000000001,"5min":9.077,"15min":15.353},"min":{"1min":7.5209999999999999,"5min":6.1050000000000004,"15min":5.3090000000000002},"max":{"1min":15.089,"5min":16.981999999999999,"15min":174.20599999999999},"last":8.0690000000000008}]},{"osd":4,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.705,"5min":11.641999999999999,"15min":19.434999999999999},"min":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.0419999999999998},"max":{"1min":18.649999999999999,"5min":19.026,"15min":290.358},"last":12.401999999999999},{"interface":"front","average":{"1min":12.657999999999999,"5min":12.154999999999999,"15min":19.594999999999999},"min":{"1min":7.8029999999999999,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":17.945,"5min":19.114000000000001,"15min":203.929},"last":15.039999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":32.521999999999998,"5min":19.515000000000001,"15min":24.640999999999998},"min":{"1min":10.952999999999999,"5min":10.938000000000001,"15min":7.7690000000000001},"max":{"1min":333.11799999999999,"5min":333.11799999999999,"15min":371.50599999999997},"last":13.135999999999999},{"interface":"front","average":{"1min":28.724,"5min":15.845000000000001,"15min":19.704000000000001},"min":{"1min":9.0389999999999997,"5min":7.6150000000000002,"15min":7.282},"max":{"1min":298.12099999999998,"5min":298.12099999999998,"15min":298.12099999999998},"last":8.6370000000000005}]},{"osd":6,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":14.052,"5min":13.733000000000001,"15min":19.829000000000001},"min":{"1min":9.0879999999999992,"5min":7.3490000000000002,"15min":7.3490000000000002},"max":{"1min":19.085000000000001,"5min":24.93,"15min":292.72300000000001},"last":16.273},{"interface":"front","average":{"1min":15.968999999999999,"5min":14.984,"15min":22.007999999999999},"min":{"1min":10.695,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":18.954999999999998,"5min":26.942,"15min":173.17599999999999},"last":14.446999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:17:02 2023","interfaces":[{"interface":"back","average":{"1min":14.409000000000001,"5min":14.747999999999999,"15min":27.568000000000001},"min":{"1min":10.997,"5min":9.3789999999999996,"15min":9.3789999999999996},"max":{"1min":17.913,"5min":19.596,"15min":350.50999999999999},"last":94.143000000000001},{"interface":"front","average":{"1min":16.77,"5min":16.920000000000002,"15min":24.178000000000001},"min":{"1min":12.005000000000001,"5min":10.826000000000001,"15min":8.3659999999999997},"max":{"1min":24.143000000000001,"5min":24.143000000000001,"15min":304.64499999999998},"last":90.555999999999997}]}]},{"osd":7,"up_from":13,"seq":55834575233,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69900,"kb_used_data":6076,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301940,"statfs":{"total":96636764160,"available":96565186560,"internally_reserved":0,"allocated":6221824,"data_stored":2925831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.7469999999999999,"5min":9.6129999999999995,"15min":18.939},"min":{"1min":5.4649999999999999,"5min":5.2800000000000002,"15min":4.2489999999999997},"max":{"1min":13.460000000000001,"5min":17.106999999999999,"15min":259.01499999999999},"last":5.3399999999999999},{"interface":"front","average":{"1min":14.004,"5min":14.007999999999999,"15min":23.878},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.016},"max":{"1min":22.023,"5min":22.023,"15min":292.73899999999998},"last":9.3879999999999999}]},{"osd":1,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.0190000000000001,"5min":9.6690000000000005,"15min":19.279},"min":{"1min":4.5,"5min":4.5,"15min":4.5},"max":{"1min":17.582999999999998,"5min":20.797000000000001,"15min":414.57100000000003},"last":12.891999999999999},{"interface":"front","average":{"1min":10.18,"5min":10.154999999999999,"15min":17.631},"min":{"1min":6.9130000000000003,"5min":6.4880000000000004,"15min":5.7709999999999999},"max":{"1min":16.693999999999999,"5min":19.157,"15min":333.53800000000001},"last":10.563000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":11.805,"5min":12.065,"15min":21.433},"min":{"1min":7.7130000000000001,"5min":6.75,"15min":6.75},"max":{"1min":18.497,"5min":19.593,"15min":286.63499999999999},"last":11.503},{"interface":"front","average":{"1min":14.807,"5min":12.464,"15min":21.170000000000002},"min":{"1min":9.2910000000000004,"5min":7.0540000000000003,"15min":6.1280000000000001},"max":{"1min":19.707999999999998,"5min":21.053000000000001,"15min":299.54300000000001},"last":15.297000000000001}]},{"osd":3,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.760999999999999,"15min":24.027000000000001},"min":{"1min":8.9309999999999992,"5min":7.0810000000000004,"15min":6.6289999999999996},"max":{"1min":16.792000000000002,"5min":22.248000000000001,"15min":391.23200000000003},"last":12.226000000000001},{"interface":"front","average":{"1min":16.058,"5min":16.571000000000002,"15min":26.021999999999998},"min":{"1min":10.843,"5min":10.125,"15min":8.3949999999999996},"max":{"1min":20.483000000000001,"5min":22.477,"15min":362.20299999999997},"last":17.472999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.838999999999999,"5min":17.010999999999999,"15min":23.010000000000002},"min":{"1min":12.465999999999999,"5min":7.6710000000000003,"15min":7.6710000000000003},"max":{"1min":20.530000000000001,"5min":22.053000000000001,"15min":293.358},"last":19.251000000000001},{"interface":"front","average":{"1min":14.430999999999999,"5min":13.443,"15min":20.056999999999999},"min":{"1min":9.4139999999999997,"5min":7.7149999999999999,"15min":6.6769999999999996},"max":{"1min":19.288,"5min":19.962,"15min":275.31799999999998},"last":16.486999999999998}]},{"osd":5,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":17.608000000000001,"5min":17.893000000000001,"15min":26.475000000000001},"min":{"1min":11.775,"5min":9.6720000000000006,"15min":7.7380000000000004},"max":{"1min":21.373000000000001,"5min":23.498000000000001,"15min":294.56400000000002},"last":18.469000000000001},{"interface":"front","average":{"1min":9.9760000000000009,"5min":11.784000000000001,"15min":22.891999999999999},"min":{"1min":6.3739999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"max":{"1min":17.087,"5min":21.634,"15min":280.83600000000001},"last":7.7839999999999998}]},{"osd":6,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.66,"5min":15.532999999999999,"15min":24.460000000000001},"min":{"1min":11.808999999999999,"5min":10.372,"15min":8.5579999999999998},"max":{"1min":21.218,"5min":22.132000000000001,"15min":289.798},"last":23.300999999999998},{"interface":"front","average":{"1min":12.567,"5min":13.994999999999999,"15min":23.161999999999999},"min":{"1min":8.3300000000000001,"5min":7.5129999999999999,"15min":7.5129999999999999},"max":{"1min":18.094000000000001,"5min":20.853999999999999,"15min":303.30900000000003},"last":13.523}]}]},{"osd":6,"up_from":13,"seq":55834575231,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86752,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94285088,"statfs":{"total":96636764160,"available":96547930112,"internally_reserved":0,"allocated":6684672,"data_stored":3185909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.628,"5min":10.327,"15min":14.347},"min":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":4.1920000000000002},"max":{"1min":15.675000000000001,"5min":18.510000000000002,"15min":168.34100000000001},"last":6.4409999999999998},{"interface":"front","average":{"1min":9.1110000000000007,"5min":8.9320000000000004,"15min":13.920999999999999},"min":{"1min":5.6600000000000001,"5min":5.2050000000000001,"15min":5.2050000000000001},"max":{"1min":15.646000000000001,"5min":19.010999999999999,"15min":172.452},"last":6.093}]},{"osd":1,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":12.702999999999999,"5min":13.539,"15min":16.152000000000001},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.1870000000000003},"max":{"1min":18.241,"5min":20.835000000000001,"15min":206.43899999999999},"last":11.98},{"interface":"front","average":{"1min":16.012,"5min":16.016999999999999,"15min":19.134},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":7.2080000000000002},"max":{"1min":35.750999999999998,"5min":35.750999999999998,"15min":155.94},"last":15.548999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:17:05 2023","interfaces":[{"interface":"back","average":{"1min":11.721,"5min":11.587999999999999,"15min":16.103000000000002},"min":{"1min":8.5380000000000003,"5min":6.508,"15min":5.8940000000000001},"max":{"1min":20.643000000000001,"5min":20.643000000000001,"15min":168.23599999999999},"last":11.254},{"interface":"front","average":{"1min":10.148,"5min":10.307,"15min":15.826000000000001},"min":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"max":{"1min":13.957000000000001,"5min":18.495999999999999,"15min":352.52800000000002},"last":7.1609999999999996}]},{"osd":3,"last update":"Fri Dec 8 00:17:39 2023","interfaces":[{"interface":"back","average":{"1min":11.417999999999999,"5min":11.669,"15min":17.757999999999999},"min":{"1min":6.867,"5min":6.867,"15min":5.8789999999999996},"max":{"1min":21.459,"5min":21.459,"15min":286.92399999999998},"last":10.349},{"interface":"front","average":{"1min":17.001000000000001,"5min":15.720000000000001,"15min":22.109000000000002},"min":{"1min":9.8889999999999993,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":36.771000000000001,"5min":36.771000000000001,"15min":265.44400000000002},"last":16.123000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:46 2023","interfaces":[{"interface":"back","average":{"1min":13.489000000000001,"5min":13.015000000000001,"15min":18.609000000000002},"min":{"1min":7.3319999999999999,"5min":6.7389999999999999,"15min":6.7389999999999999},"max":{"1min":33.700000000000003,"5min":33.700000000000003,"15min":353.452},"last":11.003},{"interface":"front","average":{"1min":17.852,"5min":16.946000000000002,"15min":23.024000000000001},"min":{"1min":9.7469999999999999,"5min":9.7469999999999999,"15min":8.8499999999999996},"max":{"1min":37.723999999999997,"5min":37.723999999999997,"15min":341.87299999999999},"last":14.942}]},{"osd":5,"last update":"Fri Dec 8 00:17:05 2023","interfaces":[{"interface":"back","average":{"1min":19.035,"5min":18.305,"15min":24.193999999999999},"min":{"1min":11.776999999999999,"5min":11.776999999999999,"15min":10.817},"max":{"1min":34.421999999999997,"5min":34.421999999999997,"15min":158.316},"last":18.122},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.954000000000001,"15min":17.834},"min":{"1min":7.8220000000000001,"5min":7.1349999999999998,"15min":6.056},"max":{"1min":19.646999999999998,"5min":20.881,"15min":157.48500000000001},"last":14.061}]},{"osd":7,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":13.949999999999999,"5min":14.15,"15min":18.186},"min":{"1min":8.5719999999999992,"5min":8.5719999999999992,"15min":8.4930000000000003},"max":{"1min":30.861000000000001,"5min":30.861000000000001,"15min":152.49600000000001},"last":12.648},{"interface":"front","average":{"1min":18.798999999999999,"5min":18.960000000000001,"15min":23.832999999999998},"min":{"1min":11.927,"5min":11.927,"15min":11.388},"max":{"1min":38.762,"5min":38.762,"15min":299.93099999999998},"last":16.920999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575231,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74660,"kb_used_data":6916,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297180,"statfs":{"total":96636764160,"available":96560312320,"internally_reserved":0,"allocated":7081984,"data_stored":3583744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":8.5069999999999997,"15min":14.414999999999999},"min":{"1min":4.4740000000000002,"5min":4.226,"15min":4.226},"max":{"1min":55.738,"5min":55.738,"15min":401.80099999999999},"last":55.738},{"interface":"front","average":{"1min":18.251000000000001,"5min":11.83,"15min":15.614000000000001},"min":{"1min":6.1360000000000001,"5min":4.9820000000000002,"15min":4.7690000000000001},"max":{"1min":83.034000000000006,"5min":83.034000000000006,"15min":139.387},"last":64.415000000000006}]},{"osd":1,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":16.733000000000001,"5min":13.003,"15min":17.408000000000001},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":5.5540000000000003},"max":{"1min":85.837999999999994,"5min":85.837999999999994,"15min":168.71299999999999},"last":27.565999999999999},{"interface":"front","average":{"1min":24.350000000000001,"5min":17.399999999999999,"15min":23.219999999999999},"min":{"1min":11.083,"5min":10.079000000000001,"15min":10.079000000000001},"max":{"1min":87.222999999999999,"5min":87.222999999999999,"15min":297.87299999999999},"last":70.337000000000003}]},{"osd":2,"last update":"Fri Dec 8 00:17:34 2023","interfaces":[{"interface":"back","average":{"1min":10.785,"5min":11.074999999999999,"15min":16.484999999999999},"min":{"1min":5.5460000000000003,"5min":5.5460000000000003,"15min":4.6980000000000004},"max":{"1min":17.513000000000002,"5min":20.484999999999999,"15min":222.297},"last":66.688999999999993},{"interface":"front","average":{"1min":16.869,"5min":12.349,"15min":17.806999999999999},"min":{"1min":7.9729999999999999,"5min":6.0209999999999999,"15min":5.7789999999999999},"max":{"1min":88.489000000000004,"5min":88.489000000000004,"15min":402.50200000000001},"last":62.433999999999997}]},{"osd":3,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.141999999999999,"5min":14.019,"15min":20.75},"min":{"1min":8,"5min":6.6630000000000003,"15min":6.5190000000000001},"max":{"1min":86.459000000000003,"5min":86.459000000000003,"15min":295.20499999999998},"last":83.376000000000005},{"interface":"front","average":{"1min":23.57,"5min":16.802,"15min":23.039999999999999},"min":{"1min":7.7939999999999996,"5min":7.7939999999999996,"15min":7.7939999999999996},"max":{"1min":95.340000000000003,"5min":95.340000000000003,"15min":154.19200000000001},"last":73.731999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":24.349,"5min":18.375,"15min":23.802},"min":{"1min":9.2729999999999997,"5min":9.2729999999999997,"15min":9.0760000000000005},"max":{"1min":84.105999999999995,"5min":84.105999999999995,"15min":337.97199999999998},"last":71.251999999999995},{"interface":"front","average":{"1min":20.994,"5min":15.461,"15min":19.864000000000001},"min":{"1min":11.005000000000001,"5min":9.0180000000000007,"15min":6.7350000000000003},"max":{"1min":67.284000000000006,"5min":67.284000000000006,"15min":169.846},"last":67.284000000000006}]},{"osd":6,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.216999999999999,"5min":17.192,"15min":19.756},"min":{"1min":9.3260000000000005,"5min":8.4109999999999996,"15min":7.4109999999999996},"max":{"1min":67.876999999999995,"5min":67.876999999999995,"15min":169.93199999999999},"last":67.876999999999995},{"interface":"front","average":{"1min":26.831,"5min":19.538,"15min":24.748999999999999},"min":{"1min":14.941000000000001,"5min":10.512,"15min":8.7810000000000006},"max":{"1min":89.180000000000007,"5min":89.180000000000007,"15min":395.5},"last":71.960999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":25.442,"5min":14.970000000000001,"15min":23.666},"min":{"1min":13.343,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":81.359999999999999,"5min":81.359999999999999,"15min":417.26299999999998},"last":68.477000000000004},{"interface":"front","average":{"1min":20.204000000000001,"5min":17.381,"15min":21.201000000000001},"min":{"1min":7.5359999999999996,"5min":7.5359999999999996,"15min":6.5780000000000003},"max":{"1min":84.742000000000004,"5min":84.742000000000004,"15min":352.34100000000001},"last":63.652999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575231,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70128,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301712,"statfs":{"total":96636764160,"available":96564953088,"internally_reserved":0,"allocated":6459392,"data_stored":3060478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.9109999999999996,"15min":15.105},"min":{"1min":4.9039999999999999,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":19.489999999999998,"5min":19.489999999999998,"15min":218.566},"last":5.375},{"interface":"front","average":{"1min":9.6110000000000007,"5min":10.023,"15min":16.123999999999999},"min":{"1min":5.4039999999999999,"5min":5.1520000000000001,"15min":4.8300000000000001},"max":{"1min":19.678999999999998,"5min":19.678999999999998,"15min":299.358},"last":6.4790000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":11.888999999999999,"15min":17.398},"min":{"1min":6.6440000000000001,"5min":6.0190000000000001,"15min":5.9249999999999998},"max":{"1min":16.940000000000001,"5min":18.978000000000002,"15min":291.13200000000001},"last":11.477},{"interface":"front","average":{"1min":11.286,"5min":10.930999999999999,"15min":14.776},"min":{"1min":5.3490000000000002,"5min":5.3490000000000002,"15min":4.9509999999999996},"max":{"1min":16.937000000000001,"5min":18.469000000000001,"15min":147.71000000000001},"last":10.048999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":11.776999999999999,"15min":17.225000000000001},"min":{"1min":7.2880000000000003,"5min":6.8819999999999997,"15min":6.1539999999999999},"max":{"1min":86.319000000000003,"5min":86.319000000000003,"15min":300.22399999999999},"last":16.667999999999999},{"interface":"front","average":{"1min":15.244,"5min":14.737,"15min":20.122},"min":{"1min":11.385,"5min":8.7119999999999997,"15min":6.7009999999999996},"max":{"1min":26.940999999999999,"5min":26.940999999999999,"15min":258.58499999999998},"last":19.100999999999999}]},{"osd":3,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.311,"5min":14.824999999999999,"15min":22.536999999999999},"min":{"1min":9.6920000000000002,"5min":7.4279999999999999,"15min":6.4219999999999997},"max":{"1min":51.329999999999998,"5min":51.329999999999998,"15min":400.41699999999997},"last":17.324000000000002},{"interface":"front","average":{"1min":15.747999999999999,"5min":13.279,"15min":19.422000000000001},"min":{"1min":9.8249999999999993,"5min":7.0460000000000003,"15min":7.0460000000000003},"max":{"1min":49.639000000000003,"5min":49.639000000000003,"15min":273.839},"last":12.162000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.038,"5min":15.727,"15min":22.605},"min":{"1min":10.035,"5min":7.4939999999999998,"15min":6.7569999999999997},"max":{"1min":33.755000000000003,"5min":33.755000000000003,"15min":397.96199999999999},"last":21.728999999999999},{"interface":"front","average":{"1min":17.222999999999999,"5min":14.757,"15min":20.649000000000001},"min":{"1min":10.176,"5min":7.0129999999999999,"15min":6.9900000000000002},"max":{"1min":59.944000000000003,"5min":59.944000000000003,"15min":291.952},"last":13.045999999999999}]},{"osd":6,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":19.376999999999999,"5min":15.308999999999999,"15min":20.122},"min":{"1min":8.5579999999999998,"5min":7.8940000000000001,"15min":6.1399999999999997},"max":{"1min":50.722999999999999,"5min":50.722999999999999,"15min":296.37},"last":20.477},{"interface":"front","average":{"1min":25.309000000000001,"5min":17.347000000000001,"15min":21.280000000000001},"min":{"1min":11.715,"5min":8.9130000000000003,"15min":7.8860000000000001},"max":{"1min":111.55,"5min":111.55,"15min":232.64099999999999},"last":13.978}]},{"osd":7,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":17.195,"5min":16.981999999999999,"15min":22.972999999999999},"min":{"1min":11.882,"5min":9.6539999999999999,"15min":7.4279999999999999},"max":{"1min":22.263999999999999,"5min":22.263999999999999,"15min":235.828},"last":25.847999999999999},{"interface":"front","average":{"1min":16.57,"5min":16.664000000000001,"15min":24.167000000000002},"min":{"1min":11.945,"5min":9.6530000000000005,"15min":9.0869999999999997},"max":{"1min":19.780000000000001,"5min":22.067,"15min":298.74599999999998},"last":27.341000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T00:18:04.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-08T00:18:04.526 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient: get_monmap_and_config 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient: build_initial_monmap 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient: monmap: 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:04.525668+0000 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:04.525668+0000 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:04.527 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 1 Processor -- start 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 1 -- start start 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient: init 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5b406bf50 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5b413f760 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd5b4145b00 0x7fd5b414d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b4145b00 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40ad4b0 con 0x7fd5b413f760 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40ae280 con 0x7fd5b406bf50 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b406c330 con 0x7fd5b4145b00 2023-12-08T00:18:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:04.530 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.525+0000 7fd5b9e76640 10 monclient(hunting): authenticate will time out at 3284.713853s 2023-12-08T00:18:04.530 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.526+0000 7fd5b2ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd5b413f760 0x7fd5b41453f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:47994/0 (socket says 172.21.15.129:47994) 2023-12-08T00:18:04.530 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.526+0000 7fd5b2ffd640 1 -- 172.21.15.129:0/3498969150 learned_addr learned my addr 172.21.15.129:0/3498969150 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:04.530 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.526+0000 7fd5b3fff640 1 --2- 172.21.15.129:0/3498969150 >> v2:172.21.15.184:3300/0 conn(0x7fd5b4145b00 0x7fd5b414d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.527+0000 7fd5b3fff640 10 monclient(hunting): get_auth_request con 0x7fd5b4145b00 auth_method 0 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.527+0000 7fd5b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.527+0000 7fd5b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.527+0000 7fd5b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b27fc640 1 -- 172.21.15.129:0/3498969150 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877824895 0 0) 0x7fd5b40ad4b0 con 0x7fd5b413f760 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b27fc640 10 monclient(hunting): _negotiate 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b27fc640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b27fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:04.531 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b27fc640 10 monclient(hunting): my global_id is 15735 2023-12-08T00:18:04.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.528+0000 7fd5b27fc640 1 -- 172.21.15.129:0/3498969150 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd59c003520 con 0x7fd5b413f760 2023-12-08T00:18:04.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 10 monclient(hunting): handle_auth_done global_id 20206 payload 274 2023-12-08T00:18:04.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 1 -- 172.21.15.129:0/3498969150 >> v1:172.21.15.184:6789/0 conn(0x7fd5b406bf50 legacy=0x7fd5b413f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:04.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 1 -- 172.21.15.129:0/3498969150 >> v1:172.21.15.129:6789/0 conn(0x7fd5b413f760 legacy=0x7fd5b41453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:04.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:04.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 1 monclient: found mon.noname-c 2023-12-08T00:18:04.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:04.533 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:04.533 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.529+0000 7fd5b3fff640 1 -- 172.21.15.129:0/3498969150 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40ae4f0 con 0x7fd5b4145b00 2023-12-08T00:18:04.535 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.533+0000 7fd5b3fff640 1 --2- 172.21.15.129:0/3498969150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b4145b00 0x7fd5b414d790 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009d90 tx=0x7fd5ac02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e80486b16256985 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b27fc640 1 -- 172.21.15.129:0/3498969150 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac02cc60 con 0x7fd5b4145b00 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b27fc640 10 monclient: dump: 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:04.539 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b27fc640 10 monclient: _finish_auth 0 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b27fc640 10 monclient: _check_auth_tickets 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b9e76640 5 monclient: authenticate success, global_id 20206 2023-12-08T00:18:04.540 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.537+0000 7fd5b9e76640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:04.541 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b27fc640 1 -- 172.21.15.129:0/3498969150 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ac02cdc0 con 0x7fd5b4145b00 2023-12-08T00:18:04.541 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:04.542 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b27fc640 1 -- 172.21.15.129:0/3498969150 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac034640 con 0x7fd5b4145b00 2023-12-08T00:18:04.542 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:04.542 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:04.542 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b27fc640 10 monclient: dump: 2023-12-08T00:18:04.542 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:04.542 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:04.542 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 4 set_mon_vals no callback set 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 10 monclient: shutdown 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 1 -- 172.21.15.129:0/3498969150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b4145b00 msgr2=0x7fd5b414d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:04.543 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/3498969150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b4145b00 0x7fd5b414d790 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009d90 tx=0x7fd5ac02beb0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 1 -- 172.21.15.129:0/3498969150 shutdown_connections 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/3498969150 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b4145b00 0x7fd5b414d790 unknown :-1 s=CLOSED pgs=4853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 1 -- 172.21.15.129:0/3498969150 >> 172.21.15.129:0/3498969150 conn(0x7fd5b40b4c20 msgr2=0x7fd5b406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 1 -- 172.21.15.129:0/3498969150 shutdown_connections 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.538+0000 7fd5b9e76640 1 -- 172.21.15.129:0/3498969150 wait complete. 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.539+0000 7fd5b9e76640 10 monclient: build_initial_monmap 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.539+0000 7fd5b9e76640 10 monclient: monmap: 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:04.545 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:04.545 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:04.545 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:04.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.539+0000 7fd5b9e76640 1 librados: starting msgr at 2023-12-08T00:18:04.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.539+0000 7fd5b9e76640 1 librados: starting objecter 2023-12-08T00:18:04.545 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.539+0000 7fd5b9e76640 1 Processor -- start 2023-12-08T00:18:04.547 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 -- start start 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 librados: setting wanted keys 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 librados: calling monclient init 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient: init 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5b406bf50 0x7fd5b41d73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b406bf50 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b413f760 0x7fd5b41404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b413f760 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:04.548 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5b4145b00 0x7fd5b4141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b4145b00 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4140a20 con 0x7fd5b4145b00 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4143490 con 0x7fd5b413f760 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4143600 con 0x7fd5b406bf50 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.540+0000 7fd5b9e76640 10 monclient(hunting): authenticate will time out at 3284.728573s 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5b4145b00 0x7fd5b4141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b413f760 0x7fd5b41404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b37fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5b406bf50 0x7fd5b41d73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b413f760 0x7fd5b41404e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:47936/0 (socket says 172.21.15.129:47936) 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 1 -- 172.21.15.129:0/1293110961 learned_addr learned my addr 172.21.15.129:0/1293110961 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:04.549 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 10 monclient(hunting): get_auth_request con 0x7fd5b413f760 auth_method 0 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b3fff640 10 monclient(hunting): get_auth_request con 0x7fd5b4145b00 auth_method 0 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b37fe640 10 monclient(hunting): get_auth_request con 0x7fd5b406bf50 auth_method 0 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b37fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:04.550 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.541+0000 7fd5b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.542+0000 7fd5b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.542+0000 7fd5b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.542+0000 7fd5b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.542+0000 7fd5b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.542+0000 7fd5b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.542+0000 7fd5b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 10 monclient(hunting): handle_auth_done global_id 20212 payload 824 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 1 -- 172.21.15.129:0/1293110961 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5b406bf50 msgr2=0x7fd5b41d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5b406bf50 0x7fd5b41d73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 1 -- 172.21.15.129:0/1293110961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5b4145b00 msgr2=0x7fd5b4141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5b4145b00 0x7fd5b4141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 10 monclient: _finish_hunting 0 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 1 monclient: found mon.noname-b 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:04.551 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:04.552 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.544+0000 7fd5b2ffd640 1 -- 172.21.15.129:0/1293110961 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac009a70 con 0x7fd5b413f760 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.546+0000 7fd5b2ffd640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b413f760 0x7fd5b41404e0 secure :-1 s=READY pgs=4854 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800c500 tx=0x7fd5a800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.546+0000 7fd59bfff640 10 client.?.objecter ms_handle_connect 0x7fd5b413f760 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.546+0000 7fd59bfff640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 1 -- 172.21.15.129:0/1293110961 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8005ce0 con 0x7fd5b413f760 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: dump: 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: _finish_auth 0 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: _check_auth_tickets 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 1 -- 172.21.15.129:0/1293110961 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a8005e40 con 0x7fd5b413f760 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 1 -- 172.21.15.129:0/1293110961 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8007330 con 0x7fd5b413f760 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 5 monclient: authenticate success, global_id 20212 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd59bfff640 10 monclient: dump: 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 monclient: _renew_subs 2023-12-08T00:18:04.555 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b41d7dc0 con 0x7fd5b413f760 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b17fa640 4 set_mon_vals no callback set 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 client.20212.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 monclient: _renew_subs 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b4141760 con 0x7fd5b413f760 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 1 librados: init done 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:04.557 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:04.557 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.551+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b4140f40 con 0x7fd5b413f760 2023-12-08T00:18:04.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.569+0000 7fd59bfff640 1 -- 172.21.15.129:0/1293110961 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd5a8007490 con 0x7fd5b413f760 2023-12-08T00:18:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.569+0000 7fd59bfff640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd5a006c8a0 0x7fd5a006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.569+0000 7fd5b37fe640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd5a006c8a0 0x7fd5a006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.569+0000 7fd5b37fe640 10 monclient: get_auth_request con 0x7fd5a006c8a0 auth_method 0 2023-12-08T00:18:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.570+0000 7fd5b37fe640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd5a006c8a0 0x7fd5a006ed60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4001780 tx=0x7fd5a4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.570+0000 7fd59bfff640 10 client.20212.objecter ms_handle_connect 0x7fd5a006c8a0 2023-12-08T00:18:04.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.572+0000 7fd59bfff640 1 -- 172.21.15.129:0/1293110961 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1046..1046 src has 978..1046) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fd5a8007740 con 0x7fd5b413f760 2023-12-08T00:18:04.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.572+0000 7fd59bfff640 10 client.20212.objecter ms_dispatch 0x7fd5b4000b90 osd_map(1046..1046 src has 978..1046) v4 2023-12-08T00:18:04.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.572+0000 7fd59bfff640 3 client.20212.objecter handle_osd_map got epochs [1046,1046] > 0 2023-12-08T00:18:04.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.572+0000 7fd59bfff640 3 client.20212.objecter handle_osd_map decoding full epoch 1046 2023-12-08T00:18:04.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.572+0000 7fd59bfff640 20 client.20212.objecter dump_active .. 0 homeless 2023-12-08T00:18:04.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.804+0000 7fd59bfff640 1 -- 172.21.15.129:0/1293110961 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5a8054110 con 0x7fd5b413f760 2023-12-08T00:18:04.806 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.804+0000 7fd59bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:04.806 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.804+0000 7fd59bfff640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:04.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.983+0000 7fd5b9e76640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-08T00:18:04.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.983+0000 7fd5b9e76640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-08T00:18:04.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.983+0000 7fd5b9e76640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:04.983+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5b4140ef0 con 0x7fd5b413f760 2023-12-08T00:18:05.020 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.019+0000 7fd59bfff640 1 -- 172.21.15.129:0/1293110961 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1046) v1 ==== 74+0+17764 (secure 0 0 0) 0x7fd5a8057730 con 0x7fd5b413f760 2023-12-08T00:18:05.021 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.019+0000 7fd59bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-08T00:18:05.021 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.019+0000 7fd59bfff640 10 monclient: _finish_command 2 = system:0 2023-12-08T00:18:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-08T00:18:05.021 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1046,"fsid":"a6eef5de-0da3-4292-875d-ef3a20551b59","created":"2023-12-07T23:45:15.852811+0000","modified":"2023-12-08T00:18:02.035246+0000","last_up_change":"2023-12-07T23:46:04.801550+0000","last_in_change":"2023-12-07T23:45:21.546459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":111,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T23:46:02.095929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T23:46:10.536846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-12-07T23:47:10.615028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd51c7b8-344d-44a8-9d6d-79c958dbaa17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1039,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6825","nonce":771847878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6827","nonce":771847878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6831","nonce":771847878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":771847878},{"type":"v1","addr":"172.21.15.129:6829","nonce":771847878}]},"public_addr":"172.21.15.129:6825/771847878","cluster_addr":"172.21.15.129:6827/771847878","heartbeat_back_addr":"172.21.15.129:6831/771847878","heartbeat_front_addr":"172.21.15.129:6829/771847878","state":["exists","up"]},{"osd":1,"uuid":"81a06079-4753-4cb3-8d07-858bf1ee6e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1037,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6817","nonce":3964479910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6819","nonce":3964479910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6823","nonce":3964479910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3964479910},{"type":"v1","addr":"172.21.15.129:6821","nonce":3964479910}]},"public_addr":"172.21.15.129:6817/3964479910","cluster_addr":"172.21.15.129:6819/3964479910","heartbeat_back_addr":"172.21.15.129:6823/3964479910","heartbeat_front_addr":"172.21.15.129:6821/3964479910","state":["exists","up"]},{"osd":2,"uuid":"b8306026-161e-40a0-bf4b-856ecf0b4d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1039,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6801","nonce":262618579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6803","nonce":262618579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6807","nonce":262618579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":262618579},{"type":"v1","addr":"172.21.15.129:6805","nonce":262618579}]},"public_addr":"172.21.15.129:6801/262618579","cluster_addr":"172.21.15.129:6803/262618579","heartbeat_back_addr":"172.21.15.129:6807/262618579","heartbeat_front_addr":"172.21.15.129:6805/262618579","state":["exists","up"]},{"osd":3,"uuid":"ebff76c4-7ecb-4c8f-9139-7c8dbe0dff19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1037,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6809","nonce":2348435352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6811","nonce":2348435352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6815","nonce":2348435352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2348435352},{"type":"v1","addr":"172.21.15.129:6813","nonce":2348435352}]},"public_addr":"172.21.15.129:6809/2348435352","cluster_addr":"172.21.15.129:6811/2348435352","heartbeat_back_addr":"172.21.15.129:6815/2348435352","heartbeat_front_addr":"172.21.15.129:6813/2348435352","state":["exists","up"]},{"osd":4,"uuid":"ce9d5aa9-cc19-4fb5-8089-5e185304551e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1039,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6827","nonce":917696104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6829","nonce":917696104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6833","nonce":917696104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":917696104},{"type":"v1","addr":"172.21.15.184:6831","nonce":917696104}]},"public_addr":"172.21.15.184:6827/917696104","cluster_addr":"172.21.15.184:6829/917696104","heartbeat_back_addr":"172.21.15.184:6833/917696104","heartbeat_front_addr":"172.21.15.184:6831/917696104","state":["exists","up"]},{"osd":5,"uuid":"714c8fce-fdd4-49b3-9cce-fd43ca74b16d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1037,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6819","nonce":4213359653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6821","nonce":4213359653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6825","nonce":4213359653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":4213359653},{"type":"v1","addr":"172.21.15.184:6823","nonce":4213359653}]},"public_addr":"172.21.15.184:6819/4213359653","cluster_addr":"172.21.15.184:6821/4213359653","heartbeat_back_addr":"172.21.15.184:6825/4213359653","heartbeat_front_addr":"172.21.15.184:6823/4213359653","state":["exists","up"]},{"osd":6,"uuid":"17d9fe56-0780-437b-b6ab-43f48471716a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1039,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6807","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6809","nonce":105483051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6813","nonce":105483051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6817","nonce":105483051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":105483051},{"type":"v1","addr":"172.21.15.184:6815","nonce":105483051}]},"public_addr":"172.21.15.184:6809/105483051","cluster_addr":"172.21.15.184:6813/105483051","heartbeat_back_addr":"172.21.15.184:6817/105483051","heartbeat_front_addr":"172.21.15.184:6815/105483051","state":["exists","up"]},{"osd":7,"uuid":"29e1d98a-ea92-42d8-b827-0004493a6525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1037,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6803","nonce":59760990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6805","nonce":59760990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6811","nonce":59760990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":59760990},{"type":"v1","addr":"172.21.15.184:6808","nonce":59760990}]},"public_addr":"172.21.15.184:6803/59760990","cluster_addr":"172.21.15.184:6805/59760990","heartbeat_back_addr":"172.21.15.184:6811/59760990","heartbeat_front_addr":"172.21.15.184:6808/59760990","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.382162+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:00.490549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.542980+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:46:01.347947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.274789+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.915445+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.493984+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T23:45:59.330992+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":14,"snaps":[{"begin":7,"length":1}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":68,"snaps":[{"begin":1,"length":1}]},{"pool":78,"snaps":[{"begin":1,"length":1}]},{"pool":88,"snaps":[{"begin":1,"length":2}]},{"pool":97,"snaps":[{"begin":3,"length":1}]},{"pool":113,"snaps":[{"begin":6,"length":2},{"begin":11,"length":2}]},{"pool":117,"snaps":[{"begin":2,"length":2}]},{"pool":131,"snaps":[{"begin":2,"length":3}]},{"pool":150,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":153,"snaps":[{"begin":2,"length":2}]},{"pool":154,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":164,"snaps":[{"begin":3,"length":1}]},{"pool":172,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":176,"snaps":[{"begin":6,"length":2}]},{"pool":194,"snaps":[{"begin":4,"length":2}]},{"pool":258,"snaps":[{"begin":12,"length":1}]},{"pool":268,"snaps":[{"begin":14,"length":1}]},{"pool":271,"snaps":[{"begin":17,"length":1}]},{"pool":272,"snaps":[{"begin":19,"length":2}]},{"pool":275,"snaps":[{"begin":24,"length":1}]},{"pool":318,"snaps":[{"begin":2,"length":1}]},{"pool":319,"snaps":[{"begin":4,"length":1}]},{"pool":322,"snaps":[{"begin":6,"length":1}]},{"pool":326,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T00:18:05.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 10 librados: watch_flush enter 2023-12-08T00:18:05.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 10 librados: watch_flush exit 2023-12-08T00:18:05.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 20 client.20212.objecter shutdown clearing up homeless session... 2023-12-08T00:18:05.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 10 client.20212.objecter successfully canceled tick 2023-12-08T00:18:05.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd5a006c8a0 msgr2=0x7fd5a006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:05.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd5a006c8a0 0x7fd5a006ed60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4001780 tx=0x7fd5a4009040 comp rx=0 tx=0).stop 2023-12-08T00:18:05.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 10 monclient: shutdown 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b413f760 msgr2=0x7fd5b41404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b413f760 0x7fd5b41404e0 secure :-1 s=READY pgs=4854 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800c500 tx=0x7fd5a800c9c0 comp rx=0 tx=0).stop 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 shutdown_connections 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd5a006c8a0 0x7fd5a006ed60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5b4145b00 0x7fd5b4141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5b413f760 0x7fd5b41404e0 unknown :-1 s=CLOSED pgs=4854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 --2- 172.21.15.129:0/1293110961 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5b406bf50 0x7fd5b41d73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 >> 172.21.15.129:0/1293110961 conn(0x7fd5b40b4c20 msgr2=0x7fd5b413f030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 shutdown_connections 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 -- 172.21.15.129:0/1293110961 wait complete. 2023-12-08T00:18:05.024 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:05.021+0000 7fd5b9e76640 1 librados: shutdown 2023-12-08T00:18:06.035 INFO:tasks.ceph:Scrubbing osd.0 2023-12-08T00:18:06.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:06.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.138+0000 7efca3fd0640 10 monclient: get_monmap_and_config 2023-12-08T00:18:06.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.138+0000 7efca3fd0640 10 monclient: build_initial_monmap 2023-12-08T00:18:06.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.138+0000 7efca3fd0640 10 monclient: monmap: 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:06.138470+0000 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:06.138470+0000 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:06.140 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.138+0000 7efca3fd0640 1 Processor -- start 2023-12-08T00:18:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 1 -- start start 2023-12-08T00:18:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient: init 2023-12-08T00:18:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-12-08T00:18:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071aa0 0x7efc9c13f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient(hunting): picked mon.noname-b con 0x7efc9c071aa0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efc9c071e80 0x7efc9c13f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient(hunting): picked mon.noname-c con 0x7efc9c071e80 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc9c142000 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c070dc0 con 0x7efc9c142000 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9c071030 con 0x7efc9c071e80 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c072b20 con 0x7efc9c071aa0 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.139+0000 7efca3fd0640 10 monclient(hunting): authenticate will time out at 3286.326695s 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca156b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efc9c071e80 0x7efc9c13f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca1d6c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071aa0 0x7efc9c13f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca156b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efc9c071e80 0x7efc9c13f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:47948/0 (socket says 172.21.15.129:47948) 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca156b640 1 -- 172.21.15.129:0/3782713518 learned_addr learned my addr 172.21.15.129:0/3782713518 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca156b640 10 monclient(hunting): get_auth_request con 0x7efc9c071e80 auth_method 0 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca156b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca156b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca156b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca1d6c640 10 monclient(hunting): get_auth_request con 0x7efc9c071aa0 auth_method 0 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca1d6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca1d6c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.140+0000 7efca1d6c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.141+0000 7efca156b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.141+0000 7efca156b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.141+0000 7efca156b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.141+0000 7efca1d6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.141+0000 7efca1d6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.141+0000 7efca1d6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 10 monclient(hunting): handle_auth_done global_id 20215 payload 274 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 1 -- 172.21.15.129:0/3782713518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071aa0 msgr2=0x7efc9c13f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 1 --2- 172.21.15.129:0/3782713518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071aa0 0x7efc9c13f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 1 -- 172.21.15.129:0/3782713518 >> v1:172.21.15.129:6789/0 conn(0x7efc9c142000 legacy=0x7efc9c142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 10 monclient: _finish_hunting 0 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 1 monclient: found mon.noname-c 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.142+0000 7efca156b640 1 -- 172.21.15.129:0/3782713518 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c0723e0 con 0x7efc9c071e80 2023-12-08T00:18:06.148 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.145+0000 7efca156b640 1 --2- 172.21.15.129:0/3782713518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c071e80 0x7efc9c13f5e0 secure :-1 s=READY pgs=4855 cs=0 l=1 rev1=1 crypto rx=0x7efc98009870 tx=0x7efc9802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8ffa33b2a702130 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 1 -- 172.21.15.129:0/3782713518 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98031640 con 0x7efc9c071e80 2023-12-08T00:18:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: dump: 2023-12-08T00:18:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: _finish_auth 0 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: _check_auth_tickets 2023-12-08T00:18:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 1 -- 172.21.15.129:0/3782713518 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc980317a0 con 0x7efc9c071e80 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 1 -- 172.21.15.129:0/3782713518 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98031b30 con 0x7efc9c071e80 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca0d6a640 10 monclient: dump: 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.154 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca3fd0640 5 monclient: authenticate success, global_id 20215 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca3fd0640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.150+0000 7efca3fd0640 4 set_mon_vals no callback set 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 10 monclient: shutdown 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 -- 172.21.15.129:0/3782713518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c071e80 msgr2=0x7efc9c13f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 --2- 172.21.15.129:0/3782713518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c071e80 0x7efc9c13f5e0 secure :-1 s=READY pgs=4855 cs=0 l=1 rev1=1 crypto rx=0x7efc98009870 tx=0x7efc9802b8f0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 -- 172.21.15.129:0/3782713518 shutdown_connections 2023-12-08T00:18:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 --2- 172.21.15.129:0/3782713518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c071e80 0x7efc9c13f5e0 unknown :-1 s=CLOSED pgs=4855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 --2- 172.21.15.129:0/3782713518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071aa0 0x7efc9c13f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 -- 172.21.15.129:0/3782713518 >> 172.21.15.129:0/3782713518 conn(0x7efc9c0b5b90 msgr2=0x7efc9c0b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 -- 172.21.15.129:0/3782713518 shutdown_connections 2023-12-08T00:18:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.151+0000 7efca3fd0640 1 -- 172.21.15.129:0/3782713518 wait complete. 2023-12-08T00:18:06.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient: build_initial_monmap 2023-12-08T00:18:06.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient: monmap: 2023-12-08T00:18:06.160 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:06.160 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:06.160 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:06.160 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 librados: starting msgr at 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 librados: starting objecter 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 Processor -- start 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 -- start start 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 librados: setting wanted keys 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 librados: calling monclient init 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient: init 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-08T00:18:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc9c071aa0 0x7efc9c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): picked mon.noname-a con 0x7efc9c071aa0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071e80 0x7efc9c1daa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): picked mon.noname-c con 0x7efc9c071e80 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c142000 0x7efc9c1dafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): picked mon.noname-b con 0x7efc9c142000 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1340b0 con 0x7efc9c071aa0 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1349d0 con 0x7efc9c142000 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1331a0 con 0x7efc9c071e80 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.152+0000 7efca3fd0640 10 monclient(hunting): authenticate will time out at 3286.339927s 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca156b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071e80 0x7efc9c1daa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca256d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c142000 0x7efc9c1dafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca256d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c142000 0x7efc9c1dafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:47950/0 (socket says 172.21.15.129:47950) 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca256d640 1 -- 172.21.15.129:0/2637623339 learned_addr learned my addr 172.21.15.129:0/2637623339 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca256d640 10 monclient(hunting): get_auth_request con 0x7efc9c142000 auth_method 0 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca256d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca256d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca256d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca156b640 10 monclient(hunting): get_auth_request con 0x7efc9c071e80 auth_method 0 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca156b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca156b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.153+0000 7efca156b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.154+0000 7efca256d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.154+0000 7efca256d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.154+0000 7efca256d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.154+0000 7efca156b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.154+0000 7efca156b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.154+0000 7efca156b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 10 monclient(hunting): handle_auth_done global_id 20218 payload 824 2023-12-08T00:18:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 1 -- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071e80 msgr2=0x7efc9c1daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efc9c071e80 0x7efc9c1daa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 1 -- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc9c071aa0 msgr2=0x7efc9c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc9c071aa0 0x7efc9c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 1 monclient: found mon.noname-b 2023-12-08T00:18:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.156+0000 7efca256d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.157+0000 7efca256d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.157+0000 7efca256d640 1 -- 172.21.15.129:0/2637623339 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc980094d0 con 0x7efc9c142000 2023-12-08T00:18:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.160+0000 7efca256d640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c142000 0x7efc9c1dafb0 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 crypto rx=0x7efc9400b990 tx=0x7efc9400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.160+0000 7efc8e7fc640 10 client.?.objecter ms_handle_connect 0x7efc9c142000 2023-12-08T00:18:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.160+0000 7efc8e7fc640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9400cba0 con 0x7efc9c142000 2023-12-08T00:18:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: dump: 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: _finish_auth 0 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: _check_auth_tickets 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9400cd00 con 0x7efc9c142000 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94014680 con 0x7efc9c142000 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8e7fc640 10 monclient: dump: 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 5 monclient: authenticate success, global_id 20218 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 10 monclient: _renew_subs 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efc8ffff640 4 set_mon_vals no callback set 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c132e30 con 0x7efc9c142000 2023-12-08T00:18:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 10 client.20218.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 10 monclient: _renew_subs 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c1db6d0 con 0x7efc9c142000 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.164+0000 7efca3fd0640 1 librados: init done 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.165+0000 7efca3fd0640 10 client.20218.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.165+0000 7efca3fd0640 15 client.20218.objecter _session_command_op_assign -1 1 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.165+0000 7efca3fd0640 20 client.20218.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.165+0000 7efca3fd0640 10 client.20218.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.165+0000 7efca3fd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.165+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc9c14c670 con 0x7efc9c142000 2023-12-08T00:18:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.186+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efc9401d3b0 con 0x7efc9c142000 2023-12-08T00:18:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.187+0000 7efc8e7fc640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efc8006c870 0x7efc8006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.187+0000 7efca1d6c640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efc8006c870 0x7efc8006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.187+0000 7efca1d6c640 10 monclient: get_auth_request con 0x7efc8006c870 auth_method 0 2023-12-08T00:18:06.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.187+0000 7efc8e7fc640 10 client.20218.objecter ms_handle_connect 0x7efc8006c870 2023-12-08T00:18:06.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.187+0000 7efca1d6c640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efc8006c870 0x7efc8006ed30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efc90006a90 tx=0x7efc90006a20 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7efc94058a10 con 0x7efc9c142000 2023-12-08T00:18:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 10 client.20218.objecter ms_dispatch 0x7efc9c000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 3 client.20218.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 3 client.20218.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 10 client.20218.objecter checking command 1 2023-12-08T00:18:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7efc80074570 0x7efc80076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 20 client.20218.objecter _get_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 20 client.20218.objecter put_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 15 client.20218.objecter _session_command_op_remove -1 1 2023-12-08T00:18:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 20 client.20218.objecter _get_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 20 client.20218.objecter get_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 15 client.20218.objecter _session_command_op_assign 0 1 2023-12-08T00:18:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 20 client.20218.objecter put_session s=0x7efc80073ea0 osd=0 4 2023-12-08T00:18:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 10 client.20218.objecter _send_command 1 2023-12-08T00:18:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 --> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc800770e0 con 0x7efc80074570 2023-12-08T00:18:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 20 client.20218.objecter dump_active .. 0 homeless 2023-12-08T00:18:06.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=1047) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc940098d0 con 0x7efc9c142000 2023-12-08T00:18:06.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.190+0000 7efc8e7fc640 10 monclient: handle_get_version_reply finishing 1 version 1047 2023-12-08T00:18:06.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.191+0000 7efca156b640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7efc80074570 0x7efc80076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.191+0000 7efca156b640 10 monclient: get_auth_request con 0x7efc80074570 auth_method 0 2023-12-08T00:18:06.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.194+0000 7efca156b640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7efc80074570 0x7efc80076a10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.194+0000 7efc8e7fc640 10 client.20218.objecter ms_handle_connect 0x7efc80074570 2023-12-08T00:18:06.218 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.217+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== osd.0 v2:172.21.15.129:6824/771847878 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7efc800770e0 con 0x7efc80074570 2023-12-08T00:18:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.217+0000 7efc8e7fc640 10 client.20218.objecter ms_dispatch 0x7efc9c000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.217+0000 7efc8e7fc640 10 client.20218.objecter _finish_command 1 = system:0 2023-12-08T00:18:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.217+0000 7efc8e7fc640 20 client.20218.objecter put_session s=0x7efc80073ea0 osd=0 4 2023-12-08T00:18:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.217+0000 7efc8e7fc640 15 client.20218.objecter _session_command_op_remove 0 1 2023-12-08T00:18:06.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 10 client.20218.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:06.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 15 client.20218.objecter _session_command_op_assign -1 2 2023-12-08T00:18:06.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 20 client.20218.objecter _get_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 20 client.20218.objecter put_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 20 client.20218.objecter _get_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 15 client.20218.objecter _session_command_op_remove -1 2 2023-12-08T00:18:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 20 client.20218.objecter get_session s=0x7efc80073ea0 osd=0 3 2023-12-08T00:18:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 15 client.20218.objecter _session_command_op_assign 0 2 2023-12-08T00:18:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 20 client.20218.objecter put_session s=0x7efc80073ea0 osd=0 4 2023-12-08T00:18:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 10 client.20218.objecter _send_command 2 2023-12-08T00:18:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.230+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 --> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efc9c134ef0 con 0x7efc80074570 2023-12-08T00:18:06.246 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efc8e7fc640 1 -- 172.21.15.129:0/2637623339 <== osd.0 v2:172.21.15.129:6824/771847878 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7efc9c134ef0 con 0x7efc80074570 2023-12-08T00:18:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efc8e7fc640 10 client.20218.objecter ms_dispatch 0x7efc9c000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efc8e7fc640 10 client.20218.objecter _finish_command 2 = system:0 2023-12-08T00:18:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efc8e7fc640 20 client.20218.objecter put_session s=0x7efc80073ea0 osd=0 4 2023-12-08T00:18:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efc8e7fc640 15 client.20218.objecter _session_command_op_remove 0 2 2023-12-08T00:18:06.247 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:06.247 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:06.247 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efca3fd0640 10 librados: watch_flush enter 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efca3fd0640 10 librados: watch_flush exit 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efca3fd0640 10 client.20218.objecter close_session for osd.0 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7efc80074570 msgr2=0x7efc80076a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.245+0000 7efca3fd0640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.129:6824/771847878,v1:172.21.15.129:6825/771847878] conn(0x7efc80074570 0x7efc80076a10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 20 client.20218.objecter put_session s=0x7efc80073ea0 osd=0 1 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 20 client.20218.objecter shutdown clearing up homeless session... 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 10 client.20218.objecter successfully canceled tick 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efc8006c870 msgr2=0x7efc8006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efc8006c870 0x7efc8006ed30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efc90006a90 tx=0x7efc90006a20 comp rx=0 tx=0).stop 2023-12-08T00:18:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 10 monclient: shutdown 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c142000 msgr2=0x7efc9c1dafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 --2- 172.21.15.129:0/2637623339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc9c142000 0x7efc9c1dafb0 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 crypto rx=0x7efc9400b990 tx=0x7efc9400be50 comp rx=0 tx=0).stop 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca256d640 1 -- 172.21.15.129:0/2637623339 reap_dead start 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 shutdown_connections 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 >> 172.21.15.129:0/2637623339 conn(0x7efc9c0b5b90 msgr2=0x7efc9c0b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 shutdown_connections 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 -- 172.21.15.129:0/2637623339 wait complete. 2023-12-08T00:18:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.246+0000 7efca3fd0640 1 librados: shutdown 2023-12-08T00:18:06.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-08T00:18:06.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.356+0000 7fb5c27f0640 10 monclient: get_monmap_and_config 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.356+0000 7fb5c27f0640 10 monclient: build_initial_monmap 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.356+0000 7fb5c27f0640 10 monclient: monmap: 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:06.356575+0000 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:06.356575+0000 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:06.359 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 1 Processor -- start 2023-12-08T00:18:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 1 -- start start 2023-12-08T00:18:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient: init 2023-12-08T00:18:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-12-08T00:18:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc06c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient(hunting): picked mon.noname-b con 0x7fb5bc06c570 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5bc0afb60 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5bc13f7d0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5bc141e20 con 0x7fb5bc0afb60 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5bc06ce90 con 0x7fb5bc13f7d0 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0709a0 con 0x7fb5bc06c570 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.357+0000 7fb5c27f0640 10 monclient(hunting): authenticate will time out at 3286.544806s 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.358+0000 7fb5c0d8d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fb5bc13f7d0 0x7fb5bc1422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:57438/0 (socket says 172.21.15.129:57438) 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.358+0000 7fb5c0d8d640 1 -- 172.21.15.129:0/3216743857 learned_addr learned my addr 172.21.15.129:0/3216743857 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.358+0000 7fb5bbfff640 1 --2- 172.21.15.129:0/3216743857 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc06c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.358+0000 7fb5bbfff640 10 monclient(hunting): get_auth_request con 0x7fb5bc06c570 auth_method 0 2023-12-08T00:18:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.358+0000 7fb5bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.358+0000 7fb5bbfff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.358+0000 7fb5bbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.359+0000 7fb5bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.359+0000 7fb5bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.359+0000 7fb5bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 1 -- 172.21.15.129:0/3216743857 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019881052 0 0) 0x7fb5bc06ce90 con 0x7fb5bc13f7d0 2023-12-08T00:18:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): _negotiate 2023-12-08T00:18:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): my global_id is 20221 2023-12-08T00:18:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 1 -- 172.21.15.129:0/3216743857 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a4003520 con 0x7fb5bc13f7d0 2023-12-08T00:18:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 1 -- 172.21.15.129:0/3216743857 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64512204 0 0) 0x7fb5bc141e20 con 0x7fb5bc0afb60 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): _negotiate 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 10 monclient(hunting): my global_id is 15741 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.360+0000 7fb5baffd640 1 -- 172.21.15.129:0/3216743857 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5bc06ce90 con 0x7fb5bc0afb60 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 10 monclient(hunting): handle_auth_done global_id 18368 payload 274 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 1 -- 172.21.15.129:0/3216743857 >> v1:172.21.15.184:6789/0 conn(0x7fb5bc13f7d0 legacy=0x7fb5bc1422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 1 -- 172.21.15.129:0/3216743857 >> v1:172.21.15.129:6789/0 conn(0x7fb5bc0afb60 legacy=0x7fb5bc13f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 1 monclient: found mon.noname-b 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-08T00:18:06.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.361+0000 7fb5bbfff640 1 -- 172.21.15.129:0/3216743857 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc06bb00 con 0x7fb5bc06c570 2023-12-08T00:18:06.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.363+0000 7fb5bbfff640 1 --2- 172.21.15.129:0/3216743857 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc06c950 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0097d0 tx=0x7fb5ac02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d71f131a15c540aa server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 1 -- 172.21.15.129:0/3216743857 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac031890 con 0x7fb5bc06c570 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 10 monclient: dump: 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.368 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 10 monclient: _finish_auth 0 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5baffd640 1 -- 172.21.15.129:0/3216743857 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac0319f0 con 0x7fb5bc06c570 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.366+0000 7fb5c27f0640 5 monclient: authenticate success, global_id 18368 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5baffd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:06.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 4 set_mon_vals no callback set 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5baffd640 1 -- 172.21.15.129:0/3216743857 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac031d80 con 0x7fb5bc06c570 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5baffd640 10 monclient: dump: 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.370 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 10 monclient: shutdown 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 1 -- 172.21.15.129:0/3216743857 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 msgr2=0x7fb5bc06c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/3216743857 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc06c950 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0097d0 tx=0x7fb5ac02b930 comp rx=0 tx=0).stop 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 1 -- 172.21.15.129:0/3216743857 shutdown_connections 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/3216743857 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc06c950 unknown :-1 s=CLOSED pgs=3480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 1 -- 172.21.15.129:0/3216743857 >> 172.21.15.129:0/3216743857 conn(0x7fb5bc066b90 msgr2=0x7fb5bc0b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 1 -- 172.21.15.129:0/3216743857 shutdown_connections 2023-12-08T00:18:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.367+0000 7fb5c27f0640 1 -- 172.21.15.129:0/3216743857 wait complete. 2023-12-08T00:18:06.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.368+0000 7fb5c27f0640 10 monclient: build_initial_monmap 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.368+0000 7fb5c27f0640 10 monclient: monmap: 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.368+0000 7fb5c27f0640 1 librados: starting msgr at 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.368+0000 7fb5c27f0640 1 librados: starting objecter 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.368+0000 7fb5c27f0640 1 Processor -- start 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 -- start start 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 librados: setting wanted keys 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 librados: calling monclient init 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient: init 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc1da940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): picked mon.noname-c con 0x7fb5bc06c570 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5bc13f7d0 0x7fb5bc12b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): picked mon.noname-a con 0x7fb5bc13f7d0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc1dae80 0x7fb5bc12b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): picked mon.noname-b con 0x7fb5bc1dae80 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1db840 con 0x7fb5bc13f7d0 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0b3360 con 0x7fb5bc1dae80 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0b5400 con 0x7fb5bc06c570 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.369+0000 7fb5c27f0640 10 monclient(hunting): authenticate will time out at 3286.556559s 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc1da940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5bc13f7d0 0x7fb5bc12b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5c0d8d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc1dae80 0x7fb5bc12b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5bc13f7d0 0x7fb5bc12b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58830/0 (socket says 172.21.15.129:58830) 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 1 -- 172.21.15.129:0/2096305358 learned_addr learned my addr 172.21.15.129:0/2096305358 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 10 monclient(hunting): get_auth_request con 0x7fb5bc13f7d0 auth_method 0 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 10 monclient(hunting): get_auth_request con 0x7fb5bc06c570 auth_method 0 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5c0d8d640 10 monclient(hunting): get_auth_request con 0x7fb5bc1dae80 auth_method 0 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5c0d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5c0d8d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5c0d8d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5c0d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.370+0000 7fb5c0d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.371+0000 7fb5c0d8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 10 monclient(hunting): handle_auth_done global_id 20224 payload 824 2023-12-08T00:18:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 1 -- 172.21.15.129:0/2096305358 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 msgr2=0x7fb5bc1da940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc1da940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 1 -- 172.21.15.129:0/2096305358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5bc13f7d0 msgr2=0x7fb5bc12b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5bc13f7d0 0x7fb5bc12b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 1 monclient: found mon.noname-b 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5c0d8d640 1 -- 172.21.15.129:0/2096305358 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac009400 con 0x7fb5bc1dae80 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.373+0000 7fb5bb7fe640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5bc13f7d0 0x7fb5bc12b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.375+0000 7fb5c0d8d640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc1dae80 0x7fb5bc12b900 secure :-1 s=READY pgs=4858 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0b67c0 tx=0x7fb5b400cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.375+0000 7fb5b8ff9640 10 client.?.objecter ms_handle_connect 0x7fb5bc1dae80 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.375+0000 7fb5b8ff9640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 1 -- 172.21.15.129:0/2096305358 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400abd0 con 0x7fb5bc1dae80 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: dump: 2023-12-08T00:18:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: _finish_auth 0 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: _check_auth_tickets 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 1 -- 172.21.15.129:0/2096305358 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b400ad30 con 0x7fb5bc1dae80 2023-12-08T00:18:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 1 -- 172.21.15.129:0/2096305358 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4012600 con 0x7fb5bc1dae80 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5b8ff9640 10 monclient: dump: 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 5 monclient: authenticate success, global_id 20224 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 10 monclient: _renew_subs 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5ba7fc640 4 set_mon_vals no callback set 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc0b4760 con 0x7fb5bc1dae80 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 10 client.20224.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 10 monclient: _renew_subs 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc0b3e90 con 0x7fb5bc1dae80 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.379+0000 7fb5c27f0640 1 librados: init done 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.380+0000 7fb5c27f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.380+0000 7fb5c27f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.380+0000 7fb5c27f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.380+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5bc04e840 con 0x7fb5bc1dae80 2023-12-08T00:18:06.620 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.618+0000 7fb5b8ff9640 1 -- 172.21.15.129:0/2096305358 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb5b4007070 con 0x7fb5bc1dae80 2023-12-08T00:18:06.620 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.619+0000 7fb5b8ff9640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb5a806c820 0x7fb5a806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.620 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.619+0000 7fb5bbfff640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb5a806c820 0x7fb5a806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.620 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.619+0000 7fb5bbfff640 10 monclient: get_auth_request con 0x7fb5a806c820 auth_method 0 2023-12-08T00:18:06.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.620+0000 7fb5bbfff640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb5a806c820 0x7fb5a806ece0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac036000 tx=0x7fb5ac03c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.621 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.620+0000 7fb5b8ff9640 10 client.20224.objecter ms_handle_connect 0x7fb5a806c820 2023-12-08T00:18:06.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.622+0000 7fb5b8ff9640 1 -- 172.21.15.129:0/2096305358 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fb5b40586d0 con 0x7fb5bc1dae80 2023-12-08T00:18:06.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.622+0000 7fb5b8ff9640 10 client.20224.objecter ms_dispatch 0x7fb5bc000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:06.623 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.622+0000 7fb5b8ff9640 3 client.20224.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:06.624 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.622+0000 7fb5b8ff9640 3 client.20224.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:06.624 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.622+0000 7fb5b8ff9640 20 client.20224.objecter dump_active .. 0 homeless 2023-12-08T00:18:06.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.632+0000 7fb5b8ff9640 1 -- 172.21.15.129:0/2096305358 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5b4054a10 con 0x7fb5bc1dae80 2023-12-08T00:18:06.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.632+0000 7fb5b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:06.633 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.632+0000 7fb5b8ff9640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:06.807 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.805+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5bc142c50 con 0x7fb5a806c820 2023-12-08T00:18:06.807 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.806+0000 7fb5b8ff9640 1 -- 172.21.15.129:0/2096305358 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb5ac031b60 con 0x7fb5a806c820 2023-12-08T00:18:06.807 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 0 to deep-scrub 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 10 librados: watch_flush enter 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 10 librados: watch_flush exit 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 20 client.20224.objecter shutdown clearing up homeless session... 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 10 client.20224.objecter successfully canceled tick 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb5a806c820 msgr2=0x7fb5a806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb5a806c820 0x7fb5a806ece0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac036000 tx=0x7fb5ac03c040 comp rx=0 tx=0).stop 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 10 monclient: shutdown 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc1dae80 msgr2=0x7fb5bc12b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc1dae80 0x7fb5bc12b900 secure :-1 s=READY pgs=4858 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0b67c0 tx=0x7fb5b400cf00 comp rx=0 tx=0).stop 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 shutdown_connections 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb5a806c820 0x7fb5a806ece0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc1dae80 0x7fb5bc12b900 unknown :-1 s=CLOSED pgs=4858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5bc13f7d0 0x7fb5bc12b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 --2- 172.21.15.129:0/2096305358 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb5bc06c570 0x7fb5bc1da940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 >> 172.21.15.129:0/2096305358 conn(0x7fb5bc066b90 msgr2=0x7fb5bc135550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 shutdown_connections 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 -- 172.21.15.129:0/2096305358 wait complete. 2023-12-08T00:18:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.808+0000 7fb5c27f0640 1 librados: shutdown 2023-12-08T00:18:06.824 INFO:tasks.ceph:Scrubbing osd.1 2023-12-08T00:18:06.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.931+0000 7fc824905640 10 monclient: get_monmap_and_config 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.931+0000 7fc824905640 10 monclient: build_initial_monmap 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.932+0000 7fc824905640 10 monclient: monmap: 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:06.932124+0000 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:06.932124+0000 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.932+0000 7fc824905640 1 Processor -- start 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.932+0000 7fc824905640 1 -- start start 2023-12-08T00:18:06.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.932+0000 7fc824905640 10 monclient: init 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc8200a8920 0x7fc8200a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): picked mon.noname-a con 0x7fc8200a8920 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc8200b20e0 0x7fc8200a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): picked mon.noname-c con 0x7fc8200b20e0 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8201459e0 0x7fc820145e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): picked mon.noname-b con 0x7fc8201459e0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.938 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8200a9e40 con 0x7fc8200a8920 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc820145590 con 0x7fc8200b20e0 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82013f100 con 0x7fc8201459e0 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc824905640 10 monclient(hunting): authenticate will time out at 3287.120330s 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc81ed9d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8201459e0 0x7fc820145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.933+0000 7fc817fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc8200b20e0 0x7fc8200a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc8200b20e0 0x7fc8200a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:47958/0 (socket says 172.21.15.129:47958) 2023-12-08T00:18:06.939 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 1 -- 172.21.15.129:0/230830432 learned_addr learned my addr 172.21.15.129:0/230830432 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 10 monclient(hunting): get_auth_request con 0x7fc8200b20e0 auth_method 0 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc81ed9d640 10 monclient(hunting): get_auth_request con 0x7fc8201459e0 auth_method 0 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc81ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc81ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc81ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc817fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc81ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.940 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc81ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.934+0000 7fc81ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 10 monclient(hunting): handle_auth_done global_id 20227 payload 274 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 1 -- 172.21.15.129:0/230830432 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8201459e0 msgr2=0x7fc820145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 1 --2- 172.21.15.129:0/230830432 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8201459e0 0x7fc820145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 1 -- 172.21.15.129:0/230830432 >> v2:172.21.15.129:3300/0 conn(0x7fc8200a8920 msgr2=0x7fc8200a8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 1 --2- 172.21.15.129:0/230830432 >> v2:172.21.15.129:3300/0 conn(0x7fc8200a8920 0x7fc8200a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 1 monclient: found mon.noname-c 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.936+0000 7fc817fff640 1 -- 172.21.15.129:0/230830432 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82013f270 con 0x7fc8200b20e0 2023-12-08T00:18:06.942 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.938+0000 7fc817fff640 1 --2- 172.21.15.129:0/230830432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8200b20e0 0x7fc8200a9900 secure :-1 s=READY pgs=4859 cs=0 l=1 rev1=1 crypto rx=0x7fc8200aaa50 tx=0x7fc80802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2baade2b0f4f616 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.941+0000 7fc81dd9b640 1 -- 172.21.15.129:0/230830432 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8080317e0 con 0x7fc8200b20e0 2023-12-08T00:18:06.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:06.944 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: dump: 2023-12-08T00:18:06.944 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.944 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: _finish_auth 0 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: _check_auth_tickets 2023-12-08T00:18:06.945 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 1 -- 172.21.15.129:0/230830432 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc808031940 con 0x7fc8200b20e0 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 5 monclient: authenticate success, global_id 20227 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 4 set_mon_vals no callback set 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 1 -- 172.21.15.129:0/230830432 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808031cd0 con 0x7fc8200b20e0 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc81dd9b640 10 monclient: dump: 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 10 monclient: shutdown 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 -- 172.21.15.129:0/230830432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8200b20e0 msgr2=0x7fc8200a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 --2- 172.21.15.129:0/230830432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8200b20e0 0x7fc8200a9900 secure :-1 s=READY pgs=4859 cs=0 l=1 rev1=1 crypto rx=0x7fc8200aaa50 tx=0x7fc80802b880 comp rx=0 tx=0).stop 2023-12-08T00:18:06.947 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 -- 172.21.15.129:0/230830432 shutdown_connections 2023-12-08T00:18:06.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 --2- 172.21.15.129:0/230830432 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8201459e0 0x7fc820145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 --2- 172.21.15.129:0/230830432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8200b20e0 0x7fc8200a9900 unknown :-1 s=CLOSED pgs=4859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 --2- 172.21.15.129:0/230830432 >> v2:172.21.15.129:3300/0 conn(0x7fc8200a8920 0x7fc8200a8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 -- 172.21.15.129:0/230830432 >> 172.21.15.129:0/230830432 conn(0x7fc820066ca0 msgr2=0x7fc8200b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:06.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 -- 172.21.15.129:0/230830432 shutdown_connections 2023-12-08T00:18:06.948 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.942+0000 7fc824905640 1 -- 172.21.15.129:0/230830432 wait complete. 2023-12-08T00:18:06.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.943+0000 7fc824905640 10 monclient: build_initial_monmap 2023-12-08T00:18:06.951 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.943+0000 7fc824905640 10 monclient: monmap: 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.943+0000 7fc824905640 1 librados: starting msgr at 2023-12-08T00:18:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.943+0000 7fc824905640 1 librados: starting objecter 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 Processor -- start 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 -- start start 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 librados: setting wanted keys 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 librados: calling monclient init 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient: init 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8200a8920 0x7fc820144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): picked mon.noname-c con 0x7fc8200a8920 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8200b20e0 0x7fc820144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): picked mon.noname-a con 0x7fc8200b20e0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8201459e0 0x7fc8201d7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): picked mon.noname-b con 0x7fc8201459e0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8201d7370 con 0x7fc8200b20e0 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc820143c40 con 0x7fc8201459e0 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc820143160 con 0x7fc8200a8920 2023-12-08T00:18:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.944+0000 7fc824905640 10 monclient(hunting): authenticate will time out at 3287.131619s 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.945+0000 7fc81e59c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8200a8920 0x7fc820144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.945+0000 7fc81e59c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8200a8920 0x7fc820144140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:58452/0 (socket says 172.21.15.129:58452) 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.945+0000 7fc81e59c640 1 -- 172.21.15.129:0/3028267142 learned_addr learned my addr 172.21.15.129:0/3028267142 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.945+0000 7fc81e59c640 10 monclient(hunting): get_auth_request con 0x7fc8200a8920 auth_method 0 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.945+0000 7fc81e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.945+0000 7fc81e59c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.945+0000 7fc81e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.946+0000 7fc81ed9d640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8201459e0 0x7fc8201d7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.946+0000 7fc81e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.955 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.946+0000 7fc81e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.946+0000 7fc81e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.947+0000 7fc81ed9d640 10 monclient(hunting): get_auth_request con 0x7fc8201459e0 auth_method 0 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.947+0000 7fc81ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.947+0000 7fc81ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.947+0000 7fc81ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.947+0000 7fc81ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.947+0000 7fc81ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.947+0000 7fc81ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 10 monclient(hunting): handle_auth_done global_id 18377 payload 824 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 1 -- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8201459e0 msgr2=0x7fc8201d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8201459e0 0x7fc8201d7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.956 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 1 -- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8200b20e0 msgr2=0x7fc820144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8200b20e0 0x7fc820144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 10 monclient: _finish_hunting 0 2023-12-08T00:18:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 1 monclient: found mon.noname-c 2023-12-08T00:18:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-08T00:18:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.950+0000 7fc81e59c640 1 -- 172.21.15.129:0/3028267142 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8080094d0 con 0x7fc8200a8920 2023-12-08T00:18:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.954+0000 7fc81e59c640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8200a8920 0x7fc820144140 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7fc81000b4a0 tx=0x7fc81000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.954+0000 7fc816ffd640 10 client.?.objecter ms_handle_connect 0x7fc8200a8920 2023-12-08T00:18:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:06.954+0000 7fc816ffd640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810010610 con 0x7fc8200a8920 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 10 monclient: dump: 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.010 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 10 monclient: _finish_auth 0 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc810010770 con 0x7fc8200a8920 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc824905640 5 monclient: authenticate success, global_id 18377 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc824905640 10 monclient: _renew_subs 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc824905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8200b57a0 con 0x7fc8200a8920 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.007+0000 7fc816ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810010b20 con 0x7fc8200a8920 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc816ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc816ffd640 10 monclient: dump: 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc81d59a640 4 set_mon_vals no callback set 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 10 client.18377.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 10 monclient: _renew_subs 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8201d7f20 con 0x7fc8200a8920 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 1 librados: init done 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 10 client.18377.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 15 client.18377.objecter _session_command_op_assign -1 1 2023-12-08T00:18:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 20 client.18377.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:07.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 10 client.18377.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:07.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:07.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.008+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc820067ba0 con 0x7fc8200a8920 2023-12-08T00:18:07.028 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.027+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc810010c80 con 0x7fc8200a8920 2023-12-08T00:18:07.029 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.028+0000 7fc816ffd640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc80006c870 0x7fc80006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.029 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.028+0000 7fc817fff640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc80006c870 0x7fc80006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.029 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.028+0000 7fc817fff640 10 monclient: get_auth_request con 0x7fc80006c870 auth_method 0 2023-12-08T00:18:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.028+0000 7fc816ffd640 10 client.18377.objecter ms_handle_connect 0x7fc80006c870 2023-12-08T00:18:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.028+0000 7fc817fff640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc80006c870 0x7fc80006ed30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc808036000 tx=0x7fc808030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.034 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.032+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fc8100584d0 con 0x7fc8200a8920 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.032+0000 7fc816ffd640 10 client.18377.objecter ms_dispatch 0x7fc820000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.032+0000 7fc816ffd640 3 client.18377.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.032+0000 7fc816ffd640 3 client.18377.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 10 client.18377.objecter checking command 1 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7fc800074570 0x7fc800076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 20 client.18377.objecter _get_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 20 client.18377.objecter put_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 15 client.18377.objecter _session_command_op_remove -1 1 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 20 client.18377.objecter _get_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 20 client.18377.objecter get_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 15 client.18377.objecter _session_command_op_assign 1 1 2023-12-08T00:18:07.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 20 client.18377.objecter put_session s=0x7fc800073ea0 osd=1 4 2023-12-08T00:18:07.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 10 client.18377.objecter _send_command 1 2023-12-08T00:18:07.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8000770e0 con 0x7fc800074570 2023-12-08T00:18:07.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc816ffd640 20 client.18377.objecter dump_active .. 0 homeless 2023-12-08T00:18:07.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc81ed9d640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7fc800074570 0x7fc800076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.033+0000 7fc81ed9d640 10 monclient: get_auth_request con 0x7fc800074570 auth_method 0 2023-12-08T00:18:07.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.036+0000 7fc816ffd640 10 client.18377.objecter ms_handle_connect 0x7fc800074570 2023-12-08T00:18:07.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.036+0000 7fc81ed9d640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7fc800074570 0x7fc800076a10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.063 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.062+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 <== osd.1 v2:172.21.15.129:6816/3964479910 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fc8000770e0 con 0x7fc800074570 2023-12-08T00:18:07.063 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.062+0000 7fc816ffd640 10 client.18377.objecter ms_dispatch 0x7fc820000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:07.063 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.062+0000 7fc816ffd640 10 client.18377.objecter _finish_command 1 = system:0 2023-12-08T00:18:07.063 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.062+0000 7fc816ffd640 20 client.18377.objecter put_session s=0x7fc800073ea0 osd=1 4 2023-12-08T00:18:07.063 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.062+0000 7fc816ffd640 15 client.18377.objecter _session_command_op_remove 1 1 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 10 client.18377.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 15 client.18377.objecter _session_command_op_assign -1 2 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 20 client.18377.objecter _get_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 20 client.18377.objecter put_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 20 client.18377.objecter _get_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 15 client.18377.objecter _session_command_op_remove -1 2 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 20 client.18377.objecter get_session s=0x7fc800073ea0 osd=1 3 2023-12-08T00:18:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 15 client.18377.objecter _session_command_op_assign 1 2 2023-12-08T00:18:07.077 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 20 client.18377.objecter put_session s=0x7fc800073ea0 osd=1 4 2023-12-08T00:18:07.077 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 10 client.18377.objecter _send_command 2 2023-12-08T00:18:07.077 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.075+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 --> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc82004e840 con 0x7fc800074570 2023-12-08T00:18:07.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.088+0000 7fc816ffd640 1 -- 172.21.15.129:0/3028267142 <== osd.1 v2:172.21.15.129:6816/3964479910 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc82004e840 con 0x7fc800074570 2023-12-08T00:18:07.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.088+0000 7fc816ffd640 10 client.18377.objecter ms_dispatch 0x7fc820000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:07.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.088+0000 7fc816ffd640 10 client.18377.objecter _finish_command 2 = system:0 2023-12-08T00:18:07.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.088+0000 7fc816ffd640 20 client.18377.objecter put_session s=0x7fc800073ea0 osd=1 4 2023-12-08T00:18:07.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.088+0000 7fc816ffd640 15 client.18377.objecter _session_command_op_remove 1 2 2023-12-08T00:18:07.090 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:07.091 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:07.091 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 10 librados: watch_flush enter 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 10 librados: watch_flush exit 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 10 client.18377.objecter close_session for osd.1 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7fc800074570 msgr2=0x7fc800076a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:6816/3964479910,v1:172.21.15.129:6817/3964479910] conn(0x7fc800074570 0x7fc800076a10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 20 client.18377.objecter put_session s=0x7fc800073ea0 osd=1 1 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 20 client.18377.objecter shutdown clearing up homeless session... 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 10 client.18377.objecter successfully canceled tick 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc80006c870 msgr2=0x7fc80006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc80006c870 0x7fc80006ed30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc808036000 tx=0x7fc808030000 comp rx=0 tx=0).stop 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 10 monclient: shutdown 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 20 monclient: shutdown canceling and discarding version request 1 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8200a8920 msgr2=0x7fc820144140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 --2- 172.21.15.129:0/3028267142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8200a8920 0x7fc820144140 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7fc81000b4a0 tx=0x7fc81000b960 comp rx=0 tx=0).stop 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc81ed9d640 1 -- 172.21.15.129:0/3028267142 reap_dead start 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 shutdown_connections 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 >> 172.21.15.129:0/3028267142 conn(0x7fc820066ca0 msgr2=0x7fc8200b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 shutdown_connections 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 -- 172.21.15.129:0/3028267142 wait complete. 2023-12-08T00:18:07.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.089+0000 7fc824905640 1 librados: shutdown 2023-12-08T00:18:07.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-08T00:18:07.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.208+0000 7f5f39432640 10 monclient: get_monmap_and_config 2023-12-08T00:18:07.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.208+0000 7f5f39432640 10 monclient: build_initial_monmap 2023-12-08T00:18:07.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.208+0000 7f5f39432640 10 monclient: monmap: 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:07.208435+0000 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:07.208435+0000 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:07.210 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 1 Processor -- start 2023-12-08T00:18:07.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 1 -- start start 2023-12-08T00:18:07.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient: init 2023-12-08T00:18:07.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:07.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5f34139a80 0x7f5f34066280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient(hunting): picked mon.noname-c con 0x7f5f34139a80 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f34139e60 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5f34143890 0x7f5f3414a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient(hunting): picked mon.noname-a con 0x7f5f34143890 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340a8d40 con 0x7f5f34139e60 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340aec30 con 0x7f5f34143890 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340ae360 con 0x7f5f34139a80 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.209+0000 7f5f39432640 10 monclient(hunting): authenticate will time out at 3287.396736s 2023-12-08T00:18:07.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f337fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5f34143890 0x7f5f3414a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f337fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5f34143890 0x7f5f3414a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58860/0 (socket says 172.21.15.129:58860) 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f337fe640 1 -- 172.21.15.129:0/1760793951 learned_addr learned my addr 172.21.15.129:0/1760793951 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f337fe640 10 monclient(hunting): get_auth_request con 0x7f5f34143890 auth_method 0 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f337fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f337fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.210+0000 7f5f32ffd640 1 --2- 172.21.15.129:0/1760793951 >> v2:172.21.15.184:3300/0 conn(0x7f5f34139a80 0x7f5f34066280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f32ffd640 10 monclient(hunting): get_auth_request con 0x7f5f34139a80 auth_method 0 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f32ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f32ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.211+0000 7f5f32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 10 monclient(hunting): handle_auth_done global_id 15747 payload 274 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 1 -- 172.21.15.129:0/1760793951 >> v2:172.21.15.184:3300/0 conn(0x7f5f34139a80 msgr2=0x7f5f34066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 1 --2- 172.21.15.129:0/1760793951 >> v2:172.21.15.184:3300/0 conn(0x7f5f34139a80 0x7f5f34066280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 1 -- 172.21.15.129:0/1760793951 >> v1:172.21.15.184:6789/0 conn(0x7f5f34139e60 legacy=0x7f5f34143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 1 monclient: found mon.noname-a 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:07.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:07.217 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.212+0000 7f5f337fe640 1 -- 172.21.15.129:0/1760793951 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340667c0 con 0x7f5f34143890 2023-12-08T00:18:07.217 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.214+0000 7f5f337fe640 1 --2- 172.21.15.129:0/1760793951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f34143890 0x7f5f3414a510 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f5f3406cbc0 tx=0x7f5f2c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62cafc0e6dd06c9 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 1 -- 172.21.15.129:0/1760793951 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c02c9f0 con 0x7f5f34143890 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: dump: 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.221 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: _finish_auth 0 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: _check_auth_tickets 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 1 -- 172.21.15.129:0/1760793951 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02cb50 con 0x7f5f34143890 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 1 -- 172.21.15.129:0/1760793951 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0343f0 con 0x7f5f34143890 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f39432640 5 monclient: authenticate success, global_id 15747 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f39432640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f39432640 4 set_mon_vals no callback set 2023-12-08T00:18:07.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.218+0000 7f5f327fc640 10 monclient: dump: 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 10 monclient: shutdown 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:07.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 -- 172.21.15.129:0/1760793951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f34143890 msgr2=0x7f5f3414a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 --2- 172.21.15.129:0/1760793951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f34143890 0x7f5f3414a510 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f5f3406cbc0 tx=0x7f5f2c02be50 comp rx=0 tx=0).stop 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 -- 172.21.15.129:0/1760793951 shutdown_connections 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 --2- 172.21.15.129:0/1760793951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f34143890 0x7f5f3414a510 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 --2- 172.21.15.129:0/1760793951 >> v2:172.21.15.184:3300/0 conn(0x7f5f34139a80 0x7f5f34066280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 -- 172.21.15.129:0/1760793951 >> 172.21.15.129:0/1760793951 conn(0x7f5f340b3cc0 msgr2=0x7f5f340b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 -- 172.21.15.129:0/1760793951 shutdown_connections 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.219+0000 7f5f39432640 1 -- 172.21.15.129:0/1760793951 wait complete. 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 10 monclient: build_initial_monmap 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 10 monclient: monmap: 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:07.224 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:07.225 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:07.225 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:07.225 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.225 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:07.225 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:07.225 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:07.225 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 1 librados: starting msgr at 2023-12-08T00:18:07.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 1 librados: starting objecter 2023-12-08T00:18:07.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 1 Processor -- start 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 1 -- start start 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 1 librados: setting wanted keys 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 1 librados: calling monclient init 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.220+0000 7f5f39432640 10 monclient: init 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f34139a80 0x7f5f341406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): picked mon.noname-b con 0x7f5f34139a80 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5f34143890 0x7f5f3413e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): picked mon.noname-c con 0x7f5f34143890 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f3413e9d0 0x7f5f341da900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.230 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): picked mon.noname-a con 0x7f5f3413e9d0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3413f2a0 con 0x7f5f3413e9d0 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3413fd30 con 0x7f5f34139a80 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3413ffa0 con 0x7f5f34143890 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f39432640 10 monclient(hunting): authenticate will time out at 3287.408186s 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f3413e9d0 0x7f5f341da900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f2bfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5f34143890 0x7f5f3413e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f3413e9d0 0x7f5f341da900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58868/0 (socket says 172.21.15.129:58868) 2023-12-08T00:18:07.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f337fe640 1 -- 172.21.15.129:0/2380200712 learned_addr learned my addr 172.21.15.129:0/2380200712 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f337fe640 10 monclient(hunting): get_auth_request con 0x7f5f3413e9d0 auth_method 0 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f337fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.221+0000 7f5f337fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request con 0x7f5f34143890 auth_method 0 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f32ffd640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f34139a80 0x7f5f341406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f32ffd640 10 monclient(hunting): get_auth_request con 0x7f5f34139a80 auth_method 0 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f32ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.222+0000 7f5f32ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.223+0000 7f5f32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.223+0000 7f5f32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.223+0000 7f5f32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 10 monclient(hunting): handle_auth_done global_id 15750 payload 824 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 1 -- 172.21.15.129:0/2380200712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5f34143890 msgr2=0x7f5f3413e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5f34143890 0x7f5f3413e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 1 -- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f34139a80 msgr2=0x7f5f341406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f34139a80 0x7f5f341406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:07.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 1 monclient: found mon.noname-a 2023-12-08T00:18:07.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:07.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:07.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.225+0000 7f5f337fe640 1 -- 172.21.15.129:0/2380200712 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c009b30 con 0x7f5f3413e9d0 2023-12-08T00:18:07.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.227+0000 7f5f337fe640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f3413e9d0 0x7f5f341da900 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009f40 tx=0x7f5f2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.227+0000 7f5f2b7fe640 10 client.?.objecter ms_handle_connect 0x7f5f3413e9d0 2023-12-08T00:18:07.234 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.227+0000 7f5f2b7fe640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 1 -- 172.21.15.129:0/2380200712 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c03e450 con 0x7f5f3413e9d0 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: dump: 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.236 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 1 -- 172.21.15.129:0/2380200712 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c03e5d0 con 0x7f5f3413e9d0 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 1 -- 172.21.15.129:0/2380200712 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c03e980 con 0x7f5f3413e9d0 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 5 monclient: authenticate success, global_id 15750 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 10 monclient: _renew_subs 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f34132820 con 0x7f5f3413e9d0 2023-12-08T00:18:07.237 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f31ffb640 4 set_mon_vals no callback set 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f2b7fe640 10 monclient: dump: 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 10 client.15750.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 10 monclient: _renew_subs 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f341dae40 con 0x7f5f3413e9d0 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.231+0000 7f5f39432640 1 librados: init done 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.232+0000 7f5f39432640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.232+0000 7f5f39432640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:07.239 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.232+0000 7f5f39432640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:07.240 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.232+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f340b0520 con 0x7f5f3413e9d0 2023-12-08T00:18:07.252 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.250+0000 7f5f2b7fe640 1 -- 172.21.15.129:0/2380200712 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5f2c03eae0 con 0x7f5f3413e9d0 2023-12-08T00:18:07.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.251+0000 7f5f2b7fe640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5f1406c870 0x7f5f1406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.251+0000 7f5f32ffd640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5f1406c870 0x7f5f1406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.251+0000 7f5f32ffd640 10 monclient: get_auth_request con 0x7f5f1406c870 auth_method 0 2023-12-08T00:18:07.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.252+0000 7f5f2b7fe640 10 client.15750.objecter ms_handle_connect 0x7f5f1406c870 2023-12-08T00:18:07.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.252+0000 7f5f32ffd640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5f1406c870 0x7f5f1406ed30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009aa0 tx=0x7f5f24009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 1 -- 172.21.15.129:0/2380200712 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f5f2c07b770 con 0x7f5f3413e9d0 2023-12-08T00:18:07.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 10 client.15750.objecter ms_dispatch 0x7f5f34000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:07.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 3 client.15750.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 3 client.15750.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 20 client.15750.objecter dump_active .. 0 homeless 2023-12-08T00:18:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 1 -- 172.21.15.129:0/2380200712 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f2c077ab0 con 0x7f5f3413e9d0 2023-12-08T00:18:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.485+0000 7f5f2b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.658+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f340b40a0 con 0x7f5f1406c870 2023-12-08T00:18:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.659+0000 7f5f2b7fe640 1 -- 172.21.15.129:0/2380200712 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5f340b40a0 con 0x7f5f1406c870 2023-12-08T00:18:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 1 to deep-scrub 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 10 librados: watch_flush enter 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 10 librados: watch_flush exit 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 20 client.15750.objecter shutdown clearing up homeless session... 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 10 client.15750.objecter successfully canceled tick 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5f1406c870 msgr2=0x7f5f1406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5f1406c870 0x7f5f1406ed30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009aa0 tx=0x7f5f24009190 comp rx=0 tx=0).stop 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 10 monclient: shutdown 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f3413e9d0 msgr2=0x7f5f341da900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f3413e9d0 0x7f5f341da900 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009f40 tx=0x7f5f2c004a40 comp rx=0 tx=0).stop 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 shutdown_connections 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f5f1406c870 0x7f5f1406ed30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f3413e9d0 0x7f5f341da900 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5f34143890 0x7f5f3413e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 --2- 172.21.15.129:0/2380200712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f34139a80 0x7f5f341406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 >> 172.21.15.129:0/2380200712 conn(0x7f5f340b3cc0 msgr2=0x7f5f34137c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 shutdown_connections 2023-12-08T00:18:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 -- 172.21.15.129:0/2380200712 wait complete. 2023-12-08T00:18:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.661+0000 7f5f39432640 1 librados: shutdown 2023-12-08T00:18:07.674 INFO:tasks.ceph:Scrubbing osd.2 2023-12-08T00:18:07.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:07.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.775+0000 7fcd0d5bb640 10 monclient: get_monmap_and_config 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.775+0000 7fcd0d5bb640 10 monclient: build_initial_monmap 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.775+0000 7fcd0d5bb640 10 monclient: monmap: 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:07.775527+0000 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:07.775527+0000 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:07.777 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 1 Processor -- start 2023-12-08T00:18:07.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 1 -- start start 2023-12-08T00:18:07.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient: init 2023-12-08T00:18:07.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:07.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-12-08T00:18:07.779 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcd080a8920 0x7fcd080a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient(hunting): picked mon.noname-a con 0x7fcd080a8920 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcd080b20e0 0x7fcd080a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient(hunting): picked mon.noname-c con 0x7fcd080b20e0 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd081459e0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0813f810 con 0x7fcd081459e0 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd080a9e40 con 0x7fcd080a8920 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd08145590 con 0x7fcd080b20e0 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.776+0000 7fcd0d5bb640 10 monclient(hunting): authenticate will time out at 3287.963726s 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd06ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcd080a8920 0x7fcd080a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd067fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcd080b20e0 0x7fcd080a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd06ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcd080a8920 0x7fcd080a8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58882/0 (socket says 172.21.15.129:58882) 2023-12-08T00:18:07.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd06ffd640 1 -- 172.21.15.129:0/1031220010 learned_addr learned my addr 172.21.15.129:0/1031220010 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd06ffd640 10 monclient(hunting): get_auth_request con 0x7fcd080a8920 auth_method 0 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd06ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.777+0000 7fcd06ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.778+0000 7fcd067fc640 10 monclient(hunting): get_auth_request con 0x7fcd080b20e0 auth_method 0 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.778+0000 7fcd067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.778+0000 7fcd067fc640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.778+0000 7fcd067fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.778+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.778+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.778+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_done global_id 15753 payload 274 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd06ffd640 1 -- 172.21.15.129:0/1031220010 >> v2:172.21.15.184:3300/0 conn(0x7fcd080b20e0 msgr2=0x7fcd080a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd06ffd640 1 --2- 172.21.15.129:0/1031220010 >> v2:172.21.15.184:3300/0 conn(0x7fcd080b20e0 0x7fcd080a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd06ffd640 1 -- 172.21.15.129:0/1031220010 >> v1:172.21.15.184:6789/0 conn(0x7fcd081459e0 legacy=0x7fcd0813f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd06ffd640 10 monclient: _finish_hunting 0 2023-12-08T00:18:07.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd06ffd640 1 monclient: found mon.noname-a 2023-12-08T00:18:07.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.779+0000 7fcd06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:07.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.780+0000 7fcd06ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:07.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.780+0000 7fcd06ffd640 1 -- 172.21.15.129:0/1031220010 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0813f9a0 con 0x7fcd080a8920 2023-12-08T00:18:07.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.781+0000 7fcd06ffd640 1 --2- 172.21.15.129:0/1031220010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd080a8920 0x7fcd080a8d00 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fcd080aaa50 tx=0x7fccf802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc03a9b878ebeb00 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 1 -- 172.21.15.129:0/1031220010 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8031870 con 0x7fcd080a8920 2023-12-08T00:18:07.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: dump: 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: _finish_auth 0 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: _check_auth_tickets 2023-12-08T00:18:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 1 -- 172.21.15.129:0/1031220010 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf80319d0 con 0x7fcd080a8920 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd0d5bb640 5 monclient: authenticate success, global_id 15753 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd0d5bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd0d5bb640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd0d5bb640 4 set_mon_vals no callback set 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 1 -- 172.21.15.129:0/1031220010 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8031d60 con 0x7fcd080a8920 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.785+0000 7fcd05ffb640 10 monclient: dump: 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.788 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.789 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 10 monclient: shutdown 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/1031220010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd080a8920 msgr2=0x7fcd080a8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 --2- 172.21.15.129:0/1031220010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd080a8920 0x7fcd080a8d00 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fcd080aaa50 tx=0x7fccf802b910 comp rx=0 tx=0).stop 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/1031220010 shutdown_connections 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 --2- 172.21.15.129:0/1031220010 >> v2:172.21.15.184:3300/0 conn(0x7fcd080b20e0 0x7fcd080a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 --2- 172.21.15.129:0/1031220010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd080a8920 0x7fcd080a8d00 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/1031220010 >> 172.21.15.129:0/1031220010 conn(0x7fcd08066ca0 msgr2=0x7fcd080b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/1031220010 shutdown_connections 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.786+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/1031220010 wait complete. 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 10 monclient: build_initial_monmap 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 10 monclient: monmap: 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.793 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 1 librados: starting msgr at 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 1 librados: starting objecter 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 1 Processor -- start 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 1 -- start start 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 1 librados: setting wanted keys 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 1 librados: calling monclient init 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.787+0000 7fcd0d5bb640 10 monclient: init 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd080a8920 0x7fcd081394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): picked mon.noname-b con 0x7fcd080a8920 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd080b20e0 0x7fcd08135b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): picked mon.noname-c con 0x7fcd080b20e0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd081459e0 0x7fcd08139ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): picked mon.noname-a con 0x7fcd081459e0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08136080 con 0x7fcd081459e0 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08133630 con 0x7fcd080a8920 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd081362f0 con 0x7fcd080b20e0 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd0d5bb640 10 monclient(hunting): authenticate will time out at 3287.975344s 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd081459e0 0x7fcd08139ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.788+0000 7fcd067fc640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd080b20e0 0x7fcd08135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd081459e0 0x7fcd08139ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58888/0 (socket says 172.21.15.129:58888) 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 1 -- 172.21.15.129:0/187145040 learned_addr learned my addr 172.21.15.129:0/187145040 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 10 monclient(hunting): get_auth_request con 0x7fcd081459e0 auth_method 0 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd067fc640 10 monclient(hunting): get_auth_request con 0x7fcd080b20e0 auth_method 0 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd067fc640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd067fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd06ffd640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd080a8920 0x7fcd081394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.789+0000 7fcd067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.790+0000 7fcd06ffd640 10 monclient(hunting): get_auth_request con 0x7fcd080a8920 auth_method 0 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.790+0000 7fcd06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.790+0000 7fcd06ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.790+0000 7fcd06ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.790+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.790+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.790+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 10 monclient(hunting): handle_auth_done global_id 20245 payload 824 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 1 -- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd080b20e0 msgr2=0x7fcd08135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd080b20e0 0x7fcd08135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 1 -- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd081459e0 msgr2=0x7fcd08139ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd081459e0 0x7fcd08139ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 10 monclient: _finish_hunting 0 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 1 monclient: found mon.noname-b 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.793+0000 7fcd06ffd640 1 -- 172.21.15.129:0/187145040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf80094d0 con 0x7fcd080a8920 2023-12-08T00:18:07.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.796+0000 7fcd06ffd640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd080a8920 0x7fcd081394d0 secure :-1 s=READY pgs=4862 cs=0 l=1 rev1=1 crypto rx=0x7fcd080b40d0 tx=0x7fccf80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.796+0000 7fccef7fe640 10 client.?.objecter ms_handle_connect 0x7fcd080a8920 2023-12-08T00:18:07.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.796+0000 7fccef7fe640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:07.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf803ac60 con 0x7fcd080a8920 2023-12-08T00:18:07.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:07.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: dump: 2023-12-08T00:18:07.801 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.801 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf803ade0 con 0x7fcd080a8920 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8044630 con 0x7fcd080a8920 2023-12-08T00:18:07.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fcd0d5bb640 5 monclient: authenticate success, global_id 20245 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fcd0d5bb640 10 monclient: _renew_subs 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fcd0d5bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0813fc20 con 0x7fcd080a8920 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fccef7fe640 10 monclient: dump: 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:07.803 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.799+0000 7fcd057fa640 4 set_mon_vals no callback set 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 10 client.20245.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 10 monclient: _renew_subs 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd081da9f0 con 0x7fcd080a8920 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 1 librados: init done 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 10 client.20245.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:07.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 15 client.20245.objecter _session_command_op_assign -1 1 2023-12-08T00:18:07.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 20 client.20245.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:07.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 10 client.20245.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:07.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:07.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.800+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd0814c6f0 con 0x7fcd080a8920 2023-12-08T00:18:07.821 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.820+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fccf804d450 con 0x7fcd080a8920 2023-12-08T00:18:07.822 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.820+0000 7fccef7fe640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fccf406c820 0x7fccf406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.822 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.821+0000 7fcd067fc640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fccf406c820 0x7fccf406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.822 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.821+0000 7fcd067fc640 10 monclient: get_auth_request con 0x7fccf406c820 auth_method 0 2023-12-08T00:18:07.822 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.821+0000 7fcd067fc640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fccf406c820 0x7fccf406ece0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcd08139ae0 tx=0x7fccfc009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.822 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.821+0000 7fccef7fe640 10 client.20245.objecter ms_handle_connect 0x7fccf406c820 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fccf802d040 con 0x7fcd080a8920 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 10 client.20245.objecter ms_dispatch 0x7fcd08000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 3 client.20245.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 3 client.20245.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 10 client.20245.objecter checking command 1 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7fccf4074520 0x7fccf40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 20 client.20245.objecter _get_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 20 client.20245.objecter put_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 15 client.20245.objecter _session_command_op_remove -1 1 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 20 client.20245.objecter _get_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 20 client.20245.objecter get_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 15 client.20245.objecter _session_command_op_assign 2 1 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 20 client.20245.objecter put_session s=0x7fccf4073e50 osd=2 4 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 10 client.20245.objecter _send_command 1 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 --> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccf4077090 con 0x7fccf4074520 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 20 client.20245.objecter dump_active .. 0 homeless 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=1047) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccf8077110 con 0x7fcd080a8920 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.822+0000 7fccef7fe640 10 monclient: handle_get_version_reply finishing 1 version 1047 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.823+0000 7fcd077fe640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7fccf4074520 0x7fccf40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.823+0000 7fcd077fe640 10 monclient: get_auth_request con 0x7fccf4074520 auth_method 0 2023-12-08T00:18:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.826+0000 7fcd077fe640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7fccf4074520 0x7fccf40769c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.826+0000 7fccef7fe640 10 client.20245.objecter ms_handle_connect 0x7fccf4074520 2023-12-08T00:18:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.849+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== osd.2 v2:172.21.15.129:6800/262618579 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fccf4077090 con 0x7fccf4074520 2023-12-08T00:18:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.849+0000 7fccef7fe640 10 client.20245.objecter ms_dispatch 0x7fcd08000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.849+0000 7fccef7fe640 10 client.20245.objecter _finish_command 1 = system:0 2023-12-08T00:18:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.849+0000 7fccef7fe640 20 client.20245.objecter put_session s=0x7fccf4073e50 osd=2 4 2023-12-08T00:18:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.849+0000 7fccef7fe640 15 client.20245.objecter _session_command_op_remove 2 1 2023-12-08T00:18:07.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 10 client.20245.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:07.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 15 client.20245.objecter _session_command_op_assign -1 2 2023-12-08T00:18:07.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 20 client.20245.objecter _get_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 20 client.20245.objecter put_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 20 client.20245.objecter _get_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.863 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 15 client.20245.objecter _session_command_op_remove -1 2 2023-12-08T00:18:07.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 20 client.20245.objecter get_session s=0x7fccf4073e50 osd=2 3 2023-12-08T00:18:07.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 15 client.20245.objecter _session_command_op_assign 2 2 2023-12-08T00:18:07.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 20 client.20245.objecter put_session s=0x7fccf4073e50 osd=2 4 2023-12-08T00:18:07.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 10 client.20245.objecter _send_command 2 2023-12-08T00:18:07.864 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.862+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 --> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcd08067ba0 con 0x7fccf4074520 2023-12-08T00:18:07.877 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.874+0000 7fccef7fe640 1 -- 172.21.15.129:0/187145040 <== osd.2 v2:172.21.15.129:6800/262618579 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcd08067ba0 con 0x7fccf4074520 2023-12-08T00:18:07.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.874+0000 7fccef7fe640 10 client.20245.objecter ms_dispatch 0x7fcd08000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:07.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.874+0000 7fccef7fe640 10 client.20245.objecter _finish_command 2 = system:0 2023-12-08T00:18:07.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.874+0000 7fccef7fe640 20 client.20245.objecter put_session s=0x7fccf4073e50 osd=2 4 2023-12-08T00:18:07.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.874+0000 7fccef7fe640 15 client.20245.objecter _session_command_op_remove 2 2 2023-12-08T00:18:07.878 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:07.878 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:07.878 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:07.879 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 10 librados: watch_flush enter 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 10 librados: watch_flush exit 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 10 client.20245.objecter close_session for osd.2 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7fccf4074520 msgr2=0x7fccf40769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.129:6800/262618579,v1:172.21.15.129:6801/262618579] conn(0x7fccf4074520 0x7fccf40769c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 20 client.20245.objecter put_session s=0x7fccf4073e50 osd=2 1 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 20 client.20245.objecter shutdown clearing up homeless session... 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 10 client.20245.objecter successfully canceled tick 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fccf406c820 msgr2=0x7fccf406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fccf406c820 0x7fccf406ece0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcd08139ae0 tx=0x7fccfc009190 comp rx=0 tx=0).stop 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 10 monclient: shutdown 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd080a8920 msgr2=0x7fcd081394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 --2- 172.21.15.129:0/187145040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd080a8920 0x7fcd081394d0 secure :-1 s=READY pgs=4862 cs=0 l=1 rev1=1 crypto rx=0x7fcd080b40d0 tx=0x7fccf80046c0 comp rx=0 tx=0).stop 2023-12-08T00:18:07.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd077fe640 1 -- 172.21.15.129:0/187145040 reap_dead start 2023-12-08T00:18:07.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 shutdown_connections 2023-12-08T00:18:07.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 >> 172.21.15.129:0/187145040 conn(0x7fcd08066ca0 msgr2=0x7fcd08073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:07.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 shutdown_connections 2023-12-08T00:18:07.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 -- 172.21.15.129:0/187145040 wait complete. 2023-12-08T00:18:07.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.875+0000 7fcd0d5bb640 1 librados: shutdown 2023-12-08T00:18:07.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-08T00:18:07.993 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.991+0000 7f6e69783640 10 monclient: get_monmap_and_config 2023-12-08T00:18:07.993 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.991+0000 7f6e69783640 10 monclient: build_initial_monmap 2023-12-08T00:18:07.993 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.991+0000 7f6e69783640 10 monclient: monmap: 2023-12-08T00:18:07.993 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:07.993 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:07.992061+0000 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:07.992061+0000 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:07.994 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:07.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.992+0000 7f6e69783640 1 Processor -- start 2023-12-08T00:18:07.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.992+0000 7f6e69783640 1 -- start start 2023-12-08T00:18:07.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.992+0000 7f6e69783640 10 monclient: init 2023-12-08T00:18:07.996 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a8950 0x7f6e6413f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient(hunting): picked mon.noname-b con 0x7f6e640a8950 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e640a9bb0 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6e6413f580 0x7f6e64145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient(hunting): picked mon.noname-c con 0x7f6e6413f580 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640aaba0 con 0x7f6e640a9bb0 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640aa9a0 con 0x7f6e6413f580 2023-12-08T00:18:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:07.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64146300 con 0x7f6e640a8950 2023-12-08T00:18:07.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:07.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e69783640 10 monclient(hunting): authenticate will time out at 3288.180365s 2023-12-08T00:18:07.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e5bfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f6e640a9bb0 0x7f6e641456b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48016/0 (socket says 172.21.15.129:48016) 2023-12-08T00:18:07.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e5bfff640 1 -- 172.21.15.129:0/4291139089 learned_addr learned my addr 172.21.15.129:0/4291139089 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.993+0000 7f6e637fe640 1 --2- 172.21.15.129:0/4291139089 >> v2:172.21.15.184:3300/0 conn(0x7f6e6413f580 0x7f6e64145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e62ffd640 1 --2- 172.21.15.129:0/4291139089 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a8950 0x7f6e6413f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e637fe640 10 monclient(hunting): get_auth_request con 0x7f6e6413f580 auth_method 0 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e637fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e637fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e62ffd640 10 monclient(hunting): get_auth_request con 0x7f6e640a8950 auth_method 0 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e62ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e62ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.994+0000 7f6e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.995+0000 7f6e62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.995+0000 7f6e62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.000 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.995+0000 7f6e62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 10 monclient(hunting): handle_auth_done global_id 20248 payload 274 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 1 -- 172.21.15.129:0/4291139089 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a8950 msgr2=0x7f6e6413f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 1 --2- 172.21.15.129:0/4291139089 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a8950 0x7f6e6413f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 1 -- 172.21.15.129:0/4291139089 >> v1:172.21.15.129:6789/0 conn(0x7f6e640a9bb0 legacy=0x7f6e641456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 1 monclient: found mon.noname-c 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.996+0000 7f6e637fe640 1 -- 172.21.15.129:0/4291139089 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640aae80 con 0x7f6e6413f580 2023-12-08T00:18:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:07.998+0000 7f6e637fe640 1 --2- 172.21.15.129:0/4291139089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64145dc0 secure :-1 s=READY pgs=4863 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009d90 tx=0x7f6e5c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35d53e182e6b371a server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 1 -- 172.21.15.129:0/4291139089 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c02cbc0 con 0x7f6e6413f580 2023-12-08T00:18:08.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:08.009 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: dump: 2023-12-08T00:18:08.009 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: _finish_auth 0 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: _check_auth_tickets 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 1 -- 172.21.15.129:0/4291139089 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c02cd20 con 0x7f6e6413f580 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:08.010 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 1 -- 172.21.15.129:0/4291139089 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c034660 con 0x7f6e6413f580 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e627fc640 10 monclient: dump: 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e69783640 5 monclient: authenticate success, global_id 20248 2023-12-08T00:18:08.011 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e69783640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.001+0000 7f6e69783640 4 set_mon_vals no callback set 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 10 monclient: shutdown 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 1 -- 172.21.15.129:0/4291139089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 msgr2=0x7f6e64145dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 1 --2- 172.21.15.129:0/4291139089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64145dc0 secure :-1 s=READY pgs=4863 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009d90 tx=0x7f6e5c02be10 comp rx=0 tx=0).stop 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 1 -- 172.21.15.129:0/4291139089 shutdown_connections 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 1 --2- 172.21.15.129:0/4291139089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64145dc0 unknown :-1 s=CLOSED pgs=4863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 1 --2- 172.21.15.129:0/4291139089 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a8950 0x7f6e6413f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.002+0000 7f6e69783640 1 -- 172.21.15.129:0/4291139089 >> 172.21.15.129:0/4291139089 conn(0x7f6e64066d40 msgr2=0x7f6e640b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.003+0000 7f6e69783640 1 -- 172.21.15.129:0/4291139089 shutdown_connections 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.003+0000 7f6e69783640 1 -- 172.21.15.129:0/4291139089 wait complete. 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.003+0000 7f6e69783640 10 monclient: build_initial_monmap 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.003+0000 7f6e69783640 10 monclient: monmap: 2023-12-08T00:18:08.012 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 librados: starting msgr at 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 librados: starting objecter 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 Processor -- start 2023-12-08T00:18:08.013 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 -- start start 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 librados: setting wanted keys 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 librados: calling monclient init 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient: init 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e640a8950 0x7f6e6413fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): picked mon.noname-a con 0x7f6e640a8950 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a9bb0 0x7f6e641305e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): picked mon.noname-c con 0x7f6e640a9bb0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64130b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): picked mon.noname-b con 0x7f6e6413f580 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:08.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64141730 con 0x7f6e640a8950 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64138530 con 0x7f6e6413f580 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64141930 con 0x7f6e640a9bb0 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.004+0000 7f6e69783640 10 monclient(hunting): authenticate will time out at 3288.191825s 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e5bfff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a9bb0 0x7f6e641305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e62ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e640a8950 0x7f6e6413fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e637fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64130b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e637fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64130b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:48032/0 (socket says 172.21.15.129:48032) 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e637fe640 1 -- 172.21.15.129:0/1418907730 learned_addr learned my addr 172.21.15.129:0/1418907730 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e637fe640 10 monclient(hunting): get_auth_request con 0x7f6e6413f580 auth_method 0 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e637fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e637fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e62ffd640 10 monclient(hunting): get_auth_request con 0x7f6e640a8950 auth_method 0 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e62ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e62ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e5bfff640 10 monclient(hunting): get_auth_request con 0x7f6e640a9bb0 auth_method 0 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e5bfff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.005+0000 7f6e5bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.006+0000 7f6e5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.018 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 10 monclient(hunting): handle_auth_done global_id 20251 payload 824 2023-12-08T00:18:08.018 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 1 -- 172.21.15.129:0/1418907730 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a9bb0 msgr2=0x7f6e641305e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.018 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a9bb0 0x7f6e641305e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.018 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 1 -- 172.21.15.129:0/1418907730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e640a8950 msgr2=0x7f6e6413fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.018 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e640a8950 0x7f6e6413fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 1 monclient: found mon.noname-b 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.009+0000 7f6e637fe640 1 -- 172.21.15.129:0/1418907730 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c009a70 con 0x7f6e6413f580 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.011+0000 7f6e637fe640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64130b20 secure :-1 s=READY pgs=4864 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009eb0 tx=0x7f6e5c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.011+0000 7f6e5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6e6413f580 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.011+0000 7f6e5b7fe640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 1 -- 172.21.15.129:0/1418907730 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c03e610 con 0x7f6e6413f580 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: dump: 2023-12-08T00:18:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 1 -- 172.21.15.129:0/1418907730 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c02ce90 con 0x7f6e6413f580 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 1 -- 172.21.15.129:0/1418907730 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c03e9b0 con 0x7f6e6413f580 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e5b7fe640 10 monclient: dump: 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 5 monclient: authenticate success, global_id 20251 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 10 monclient: _renew_subs 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e61ffb640 4 set_mon_vals no callback set 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e641389c0 con 0x7f6e6413f580 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 10 client.20251.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 10 monclient: _renew_subs 2023-12-08T00:18:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e64131860 con 0x7f6e6413f580 2023-12-08T00:18:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.015+0000 7f6e69783640 1 librados: init done 2023-12-08T00:18:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.016+0000 7f6e69783640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.016+0000 7f6e69783640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.016+0000 7f6e69783640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.016+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e641386e0 con 0x7f6e6413f580 2023-12-08T00:18:08.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.036+0000 7f6e5b7fe640 1 -- 172.21.15.129:0/1418907730 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6e5c03b050 con 0x7f6e6413f580 2023-12-08T00:18:08.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.036+0000 7f6e5b7fe640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f6e4406c820 0x7f6e4406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.037+0000 7f6e62ffd640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f6e4406c820 0x7f6e4406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.038 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.037+0000 7f6e62ffd640 10 monclient: get_auth_request con 0x7f6e4406c820 auth_method 0 2023-12-08T00:18:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.037+0000 7f6e62ffd640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f6e4406c820 0x7f6e4406ece0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009b40 tx=0x7f6e54009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.037+0000 7f6e5b7fe640 10 client.20251.objecter ms_handle_connect 0x7f6e4406c820 2023-12-08T00:18:08.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.261+0000 7f6e5b7fe640 1 -- 172.21.15.129:0/1418907730 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f6e5c07ad60 con 0x7f6e6413f580 2023-12-08T00:18:08.262 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.261+0000 7f6e5b7fe640 10 client.20251.objecter ms_dispatch 0x7f6e64000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:08.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.261+0000 7f6e5b7fe640 3 client.20251.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:08.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.261+0000 7f6e5b7fe640 3 client.20251.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:08.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.261+0000 7f6e5b7fe640 20 client.20251.objecter dump_active .. 0 homeless 2023-12-08T00:18:08.272 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.271+0000 7f6e5b7fe640 1 -- 172.21.15.129:0/1418907730 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e5c0770a0 con 0x7f6e6413f580 2023-12-08T00:18:08.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.271+0000 7f6e5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:08.273 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.271+0000 7f6e5b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:08.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.448+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e640a8f40 con 0x7f6e4406c820 2023-12-08T00:18:08.450 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.448+0000 7f6e5b7fe640 1 -- 172.21.15.129:0/1418907730 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6e54011a60 con 0x7f6e4406c820 2023-12-08T00:18:08.450 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 2 to deep-scrub 2023-12-08T00:18:08.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 10 librados: watch_flush enter 2023-12-08T00:18:08.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 10 librados: watch_flush exit 2023-12-08T00:18:08.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 20 client.20251.objecter shutdown clearing up homeless session... 2023-12-08T00:18:08.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 10 client.20251.objecter successfully canceled tick 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f6e4406c820 msgr2=0x7f6e4406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f6e4406c820 0x7f6e4406ece0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009b40 tx=0x7f6e54009190 comp rx=0 tx=0).stop 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 10 monclient: shutdown 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 msgr2=0x7f6e64130b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64130b20 secure :-1 s=READY pgs=4864 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009eb0 tx=0x7f6e5c004a40 comp rx=0 tx=0).stop 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 shutdown_connections 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f6e4406c820 0x7f6e4406ece0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6413f580 0x7f6e64130b20 unknown :-1 s=CLOSED pgs=4864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e640a9bb0 0x7f6e641305e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 --2- 172.21.15.129:0/1418907730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e640a8950 0x7f6e6413fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 >> 172.21.15.129:0/1418907730 conn(0x7f6e64066d40 msgr2=0x7f6e64138e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 shutdown_connections 2023-12-08T00:18:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 -- 172.21.15.129:0/1418907730 wait complete. 2023-12-08T00:18:08.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.450+0000 7f6e69783640 1 librados: shutdown 2023-12-08T00:18:08.464 INFO:tasks.ceph:Scrubbing osd.3 2023-12-08T00:18:08.464 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.566+0000 7f62848c2640 10 monclient: get_monmap_and_config 2023-12-08T00:18:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.566+0000 7f62848c2640 10 monclient: build_initial_monmap 2023-12-08T00:18:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.566+0000 7f62848c2640 10 monclient: monmap: 2023-12-08T00:18:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:08.566203+0000 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:08.566203+0000 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:08.568 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.566+0000 7f62848c2640 1 Processor -- start 2023-12-08T00:18:08.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.566+0000 7f62848c2640 1 -- start start 2023-12-08T00:18:08.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.566+0000 7f62848c2640 10 monclient: init 2023-12-08T00:18:08.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62800a8920 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f62800b20e0 0x7f62800a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient(hunting): picked mon.noname-a con 0x7f62800b20e0 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62800a9900 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62800a9d50 con 0x7f62800a9900 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62801460f0 con 0x7f62800a8920 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280145590 con 0x7f62800b20e0 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.567+0000 7f62848c2640 10 monclient(hunting): authenticate will time out at 3288.754498s 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627e59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f62800a8920 0x7f62801459e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:57450/0 (socket says 172.21.15.129:57450) 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627e59c640 1 -- 172.21.15.129:0/3167748611 learned_addr learned my addr 172.21.15.129:0/3167748611 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:08.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 1 --2- 172.21.15.129:0/3167748611 >> v2:172.21.15.129:3300/0 conn(0x7f62800b20e0 0x7f62800a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 10 monclient(hunting): get_auth_request con 0x7f62800b20e0 auth_method 0 2023-12-08T00:18:08.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.568+0000 7f627dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627d59a640 1 -- 172.21.15.129:0/3167748611 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246356894 0 0) 0x7f62801460f0 con 0x7f62800a8920 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627d59a640 10 monclient(hunting): _negotiate 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627d59a640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627d59a640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627d59a640 10 monclient(hunting): my global_id is 20254 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627d59a640 1 -- 172.21.15.129:0/3167748611 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626c003520 con 0x7f62800a8920 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 10 monclient(hunting): handle_auth_done global_id 15765 payload 274 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 1 -- 172.21.15.129:0/3167748611 >> v1:172.21.15.184:6789/0 conn(0x7f62800a8920 legacy=0x7f62801459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 1 -- 172.21.15.129:0/3167748611 >> v1:172.21.15.129:6789/0 conn(0x7f62800a9900 legacy=0x7f628013f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:08.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 10 monclient: _finish_hunting 0 2023-12-08T00:18:08.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 1 monclient: found mon.noname-a 2023-12-08T00:18:08.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:08.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:08.573 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.570+0000 7f627dd9b640 1 -- 172.21.15.129:0/3167748611 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280146280 con 0x7f62800b20e0 2023-12-08T00:18:08.574 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.572+0000 7f627dd9b640 1 --2- 172.21.15.129:0/3167748611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800b20e0 0x7f62800a8d00 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f62680097a0 tx=0x7f626802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9187adee80fbf5 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 1 -- 172.21.15.129:0/3167748611 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626802e070 con 0x7f62800b20e0 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: dump: 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.580 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: _finish_auth 0 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: _check_auth_tickets 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 1 -- 172.21.15.129:0/3167748611 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62680315a0 con 0x7f62800b20e0 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 1 -- 172.21.15.129:0/3167748611 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268031930 con 0x7f62800b20e0 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f627d59a640 10 monclient: dump: 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.581 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f62848c2640 5 monclient: authenticate success, global_id 15765 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f62848c2640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.576+0000 7f62848c2640 4 set_mon_vals no callback set 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 10 monclient: shutdown 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 1 -- 172.21.15.129:0/3167748611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800b20e0 msgr2=0x7f62800a8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 1 --2- 172.21.15.129:0/3167748611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800b20e0 0x7f62800a8d00 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f62680097a0 tx=0x7f626802b840 comp rx=0 tx=0).stop 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 1 -- 172.21.15.129:0/3167748611 shutdown_connections 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 1 --2- 172.21.15.129:0/3167748611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800b20e0 0x7f62800a8d00 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 1 -- 172.21.15.129:0/3167748611 >> 172.21.15.129:0/3167748611 conn(0x7f6280066ca0 msgr2=0x7f62800b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:08.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 1 -- 172.21.15.129:0/3167748611 shutdown_connections 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.577+0000 7f62848c2640 1 -- 172.21.15.129:0/3167748611 wait complete. 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 10 monclient: build_initial_monmap 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 10 monclient: monmap: 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:08.583 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 1 librados: starting msgr at 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 1 librados: starting objecter 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 1 Processor -- start 2023-12-08T00:18:08.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 1 -- start start 2023-12-08T00:18:08.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 1 librados: setting wanted keys 2023-12-08T00:18:08.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 1 librados: calling monclient init 2023-12-08T00:18:08.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.578+0000 7f62848c2640 10 monclient: init 2023-12-08T00:18:08.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:08.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-08T00:18:08.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62800a8920 0x7f6280140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): picked mon.noname-c con 0x7f62800a8920 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800a9900 0x7f6280136c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): picked mon.noname-a con 0x7f62800a9900 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62800b20e0 0x7f6280138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): picked mon.noname-b con 0x7f62800b20e0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801413b0 con 0x7f62800a9900 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6280142500 con 0x7f62800b20e0 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280137ad0 con 0x7f62800a8920 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f62848c2640 10 monclient(hunting): authenticate will time out at 3288.766432s 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f627dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800a9900 0x7f6280136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.579+0000 7f627e59c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62800a8920 0x7f6280140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800a9900 0x7f6280136c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58924/0 (socket says 172.21.15.129:58924) 2023-12-08T00:18:08.589 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 1 -- 172.21.15.129:0/1102612912 learned_addr learned my addr 172.21.15.129:0/1102612912 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:08.589 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 10 monclient(hunting): get_auth_request con 0x7f62800a9900 auth_method 0 2023-12-08T00:18:08.589 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.589 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62800b20e0 0x7f6280138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627e59c640 10 monclient(hunting): get_auth_request con 0x7f62800a8920 auth_method 0 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627e59c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 10 monclient(hunting): get_auth_request con 0x7f62800b20e0 auth_method 0 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.590 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.580+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 10 monclient(hunting): handle_auth_done global_id 20257 payload 824 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 1 -- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62800a8920 msgr2=0x7f6280140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62800a8920 0x7f6280140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 1 -- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800a9900 msgr2=0x7f6280136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62800a9900 0x7f6280136c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 1 monclient: found mon.noname-b 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.583+0000 7f627ed9d640 1 -- 172.21.15.129:0/1102612912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268009400 con 0x7f62800b20e0 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.585+0000 7f627ed9d640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62800b20e0 0x7f6280138790 secure :-1 s=READY pgs=4866 cs=0 l=1 rev1=1 crypto rx=0x7f627800c7b0 tx=0x7f627800cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.585+0000 7f6276ffd640 10 client.?.objecter ms_handle_connect 0x7f62800b20e0 2023-12-08T00:18:08.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.585+0000 7f6276ffd640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:08.593 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278003b70 con 0x7f62800b20e0 2023-12-08T00:18:08.593 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: dump: 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: _finish_auth 0 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278003cf0 con 0x7f62800b20e0 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 5 monclient: authenticate success, global_id 20257 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 monclient: _renew_subs 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6280137db0 con 0x7f62800b20e0 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62780052b0 con 0x7f62800b20e0 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f6276ffd640 10 monclient: dump: 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f627cd99640 4 set_mon_vals no callback set 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 client.20257.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 monclient: _renew_subs 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6280137170 con 0x7f62800b20e0 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 1 librados: init done 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 client.20257.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 15 client.20257.objecter _session_command_op_assign -1 1 2023-12-08T00:18:08.596 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 20 client.20257.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 client.20257.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.589+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6280067ba0 con 0x7f62800b20e0 2023-12-08T00:18:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.608+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6278005410 con 0x7f62800b20e0 2023-12-08T00:18:08.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.608+0000 7f6276ffd640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f626006c820 0x7f626006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.609+0000 7f627e59c640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f626006c820 0x7f626006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.609+0000 7f627e59c640 10 monclient: get_auth_request con 0x7f626006c820 auth_method 0 2023-12-08T00:18:08.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.609+0000 7f6276ffd640 10 client.20257.objecter ms_handle_connect 0x7f626006c820 2023-12-08T00:18:08.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.609+0000 7f627e59c640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f626006c820 0x7f626006ece0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6270005880 tx=0x7f6270009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.611+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f6278058310 con 0x7f62800b20e0 2023-12-08T00:18:08.613 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.611+0000 7f6276ffd640 10 client.20257.objecter ms_dispatch 0x7f6280000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.611+0000 7f6276ffd640 3 client.20257.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.611+0000 7f6276ffd640 3 client.20257.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.611+0000 7f6276ffd640 10 client.20257.objecter checking command 1 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.611+0000 7f6276ffd640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7f6260074520 0x7f62600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 20 client.20257.objecter _get_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 20 client.20257.objecter put_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 15 client.20257.objecter _session_command_op_remove -1 1 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 20 client.20257.objecter _get_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 20 client.20257.objecter get_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 15 client.20257.objecter _session_command_op_assign 3 1 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 20 client.20257.objecter put_session s=0x7f6260073e50 osd=3 4 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 10 client.20257.objecter _send_command 1 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 --> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6260077090 con 0x7f6260074520 2023-12-08T00:18:08.614 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 20 client.20257.objecter dump_active .. 0 homeless 2023-12-08T00:18:08.615 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=1047) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6278054650 con 0x7f62800b20e0 2023-12-08T00:18:08.615 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f6276ffd640 10 monclient: handle_get_version_reply finishing 1 version 1047 2023-12-08T00:18:08.615 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f627dd9b640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7f6260074520 0x7f62600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.615 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.612+0000 7f627dd9b640 10 monclient: get_auth_request con 0x7f6260074520 auth_method 0 2023-12-08T00:18:08.616 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.615+0000 7f627dd9b640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7f6260074520 0x7f62600769c0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.616 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.615+0000 7f6276ffd640 10 client.20257.objecter ms_handle_connect 0x7f6260074520 2023-12-08T00:18:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.638+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== osd.3 v2:172.21.15.129:6808/2348435352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f6260077090 con 0x7f6260074520 2023-12-08T00:18:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.638+0000 7f6276ffd640 10 client.20257.objecter ms_dispatch 0x7f6280000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.638+0000 7f6276ffd640 10 client.20257.objecter _finish_command 1 = system:0 2023-12-08T00:18:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.638+0000 7f6276ffd640 20 client.20257.objecter put_session s=0x7f6260073e50 osd=3 4 2023-12-08T00:18:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.638+0000 7f6276ffd640 15 client.20257.objecter _session_command_op_remove 3 1 2023-12-08T00:18:08.652 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 10 client.20257.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 15 client.20257.objecter _session_command_op_assign -1 2 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 20 client.20257.objecter _get_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 20 client.20257.objecter put_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 20 client.20257.objecter _get_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 15 client.20257.objecter _session_command_op_remove -1 2 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 20 client.20257.objecter get_session s=0x7f6260073e50 osd=3 3 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 15 client.20257.objecter _session_command_op_assign 3 2 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 20 client.20257.objecter put_session s=0x7f6260073e50 osd=3 4 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 10 client.20257.objecter _send_command 2 2023-12-08T00:18:08.653 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.651+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 --> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6280138f20 con 0x7f6260074520 2023-12-08T00:18:08.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f6276ffd640 1 -- 172.21.15.129:0/1102612912 <== osd.3 v2:172.21.15.129:6808/2348435352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6280138f20 con 0x7f6260074520 2023-12-08T00:18:08.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f6276ffd640 10 client.20257.objecter ms_dispatch 0x7f6280000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f6276ffd640 10 client.20257.objecter _finish_command 2 = system:0 2023-12-08T00:18:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f6276ffd640 20 client.20257.objecter put_session s=0x7f6260073e50 osd=3 4 2023-12-08T00:18:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f6276ffd640 15 client.20257.objecter _session_command_op_remove 3 2 2023-12-08T00:18:08.666 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:08.666 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:08.666 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 10 librados: watch_flush enter 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 10 librados: watch_flush exit 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 10 client.20257.objecter close_session for osd.3 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7f6260074520 msgr2=0x7f62600769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.129:6808/2348435352,v1:172.21.15.129:6809/2348435352] conn(0x7f6260074520 0x7f62600769c0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 20 client.20257.objecter put_session s=0x7f6260073e50 osd=3 1 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 20 client.20257.objecter shutdown clearing up homeless session... 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 10 client.20257.objecter successfully canceled tick 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f626006c820 msgr2=0x7f626006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f626006c820 0x7f626006ece0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6270005880 tx=0x7f6270009040 comp rx=0 tx=0).stop 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 10 monclient: shutdown 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62800b20e0 msgr2=0x7f6280138790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 --2- 172.21.15.129:0/1102612912 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62800b20e0 0x7f6280138790 secure :-1 s=READY pgs=4866 cs=0 l=1 rev1=1 crypto rx=0x7f627800c7b0 tx=0x7f627800cc70 comp rx=0 tx=0).stop 2023-12-08T00:18:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f627ed9d640 1 -- 172.21.15.129:0/1102612912 reap_dead start 2023-12-08T00:18:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 shutdown_connections 2023-12-08T00:18:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 >> 172.21.15.129:0/1102612912 conn(0x7f6280066ca0 msgr2=0x7f6280135610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.664+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 shutdown_connections 2023-12-08T00:18:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.665+0000 7f62848c2640 1 -- 172.21.15.129:0/1102612912 wait complete. 2023-12-08T00:18:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.665+0000 7f62848c2640 1 librados: shutdown 2023-12-08T00:18:08.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-08T00:18:08.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.776+0000 7f393a3b6640 10 monclient: get_monmap_and_config 2023-12-08T00:18:08.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.776+0000 7f393a3b6640 10 monclient: build_initial_monmap 2023-12-08T00:18:08.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.776+0000 7f393a3b6640 10 monclient: monmap: 2023-12-08T00:18:08.777 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:08.776203+0000 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:08.776203+0000 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:08.778 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.776+0000 7f393a3b6640 1 Processor -- start 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.776+0000 7f393a3b6640 1 -- start start 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.776+0000 7f393a3b6640 10 monclient: init 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39340a8970 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39340a9bd0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393413f580 0x7f39341428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f393413f580 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340aa950 con 0x7f39340a8970 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340a90e0 con 0x7f39340a9bd0 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.780 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3934142df0 con 0x7f393413f580 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f393a3b6640 10 monclient(hunting): authenticate will time out at 3288.964467s 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.777+0000 7f3933fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f39340a8970 0x7f393413ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48046/0 (socket says 172.21.15.129:48046) 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.778+0000 7f3933fff640 1 -- 172.21.15.129:0/1747818438 learned_addr learned my addr 172.21.15.129:0/1747818438 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.778+0000 7f3938953640 1 --2- 172.21.15.129:0/1747818438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393413f580 0x7f39341428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.778+0000 7f3938953640 10 monclient(hunting): get_auth_request con 0x7f393413f580 auth_method 0 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.778+0000 7f3938953640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.778+0000 7f3938953640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.778+0000 7f3938953640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3938953640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3938953640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3938953640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3932ffd640 1 -- 172.21.15.129:0/1747818438 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910759577 0 0) 0x7f39340a90e0 con 0x7f39340a9bd0 2023-12-08T00:18:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3932ffd640 10 monclient(hunting): _negotiate 2023-12-08T00:18:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3932ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3932ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3932ffd640 10 monclient(hunting): my global_id is 20260 2023-12-08T00:18:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.779+0000 7f3932ffd640 1 -- 172.21.15.129:0/1747818438 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3920003520 con 0x7f39340a9bd0 2023-12-08T00:18:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 10 monclient(hunting): handle_auth_done global_id 18395 payload 274 2023-12-08T00:18:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 1 -- 172.21.15.129:0/1747818438 >> v1:172.21.15.184:6789/0 conn(0x7f39340a9bd0 legacy=0x7f39341421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 1 -- 172.21.15.129:0/1747818438 >> v1:172.21.15.129:6789/0 conn(0x7f39340a8970 legacy=0x7f393413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 10 monclient: _finish_hunting 0 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 1 monclient: found mon.noname-b 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3932ffd640 1 -- 172.21.15.129:0/1747818438 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299401421 0 0) 0x7f39340aa950 con 0x7f39340a8970 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3938953640 1 -- 172.21.15.129:0/1747818438 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340aabc0 con 0x7f393413f580 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.781+0000 7f3932ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-08T00:18:08.786 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.783+0000 7f3938953640 1 --2- 172.21.15.129:0/1747818438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393413f580 0x7f39341428b0 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 crypto rx=0x7f392c009dc0 tx=0x7f392c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77d0fe6b949c50b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 1 -- 172.21.15.129:0/1747818438 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c02cbb0 con 0x7f393413f580 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: dump: 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: _finish_auth 0 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 1 -- 172.21.15.129:0/1747818438 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c02cd10 con 0x7f393413f580 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 1 -- 172.21.15.129:0/1747818438 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c034660 con 0x7f393413f580 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f393a3b6640 5 monclient: authenticate success, global_id 18395 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f393a3b6640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f393a3b6640 4 set_mon_vals no callback set 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:08.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.786+0000 7f3932ffd640 10 monclient: dump: 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 10 monclient: shutdown 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 1 -- 172.21.15.129:0/1747818438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393413f580 msgr2=0x7f39341428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1747818438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393413f580 0x7f39341428b0 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 crypto rx=0x7f392c009dc0 tx=0x7f392c02be00 comp rx=0 tx=0).stop 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 1 -- 172.21.15.129:0/1747818438 shutdown_connections 2023-12-08T00:18:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1747818438 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393413f580 0x7f39341428b0 unknown :-1 s=CLOSED pgs=3482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 1 -- 172.21.15.129:0/1747818438 >> 172.21.15.129:0/1747818438 conn(0x7f3934066dd0 msgr2=0x7f39340b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:08.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 1 -- 172.21.15.129:0/1747818438 shutdown_connections 2023-12-08T00:18:08.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.787+0000 7f393a3b6640 1 -- 172.21.15.129:0/1747818438 wait complete. 2023-12-08T00:18:08.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 10 monclient: build_initial_monmap 2023-12-08T00:18:08.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 10 monclient: monmap: 2023-12-08T00:18:08.794 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 1 librados: starting msgr at 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 1 librados: starting objecter 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 1 Processor -- start 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 1 -- start start 2023-12-08T00:18:08.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 1 librados: setting wanted keys 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 1 librados: calling monclient init 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.788+0000 7f393a3b6640 10 monclient: init 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39340a8970 0x7f393413df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f39340a8970 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39340a9bd0 0x7f39341da220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): picked mon.noname-c con 0x7f39340a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393413f580 0x7f39341da760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): picked mon.noname-a con 0x7f393413f580 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39341daf70 con 0x7f393413f580 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39341d6db0 con 0x7f39340a8970 2023-12-08T00:18:08.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39341d6f70 con 0x7f39340a9bd0 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f393a3b6640 10 monclient(hunting): authenticate will time out at 3288.976397s 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f3938953640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393413f580 0x7f39341da760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.789+0000 7f39337fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39340a9bd0 0x7f39341da220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393413f580 0x7f39341da760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58936/0 (socket says 172.21.15.129:58936) 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 1 -- 172.21.15.129:0/1394425300 learned_addr learned my addr 172.21.15.129:0/1394425300 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 10 monclient(hunting): get_auth_request con 0x7f393413f580 auth_method 0 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f39337fe640 10 monclient(hunting): get_auth_request con 0x7f39340a9bd0 auth_method 0 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f39337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f39337fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.797 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f39337fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3933fff640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39340a8970 0x7f393413df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3938953640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3933fff640 10 monclient(hunting): get_auth_request con 0x7f39340a8970 auth_method 0 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3933fff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f3933fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f39337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f39337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.790+0000 7f39337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.791+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.791+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:08.798 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.791+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 10 monclient(hunting): handle_auth_done global_id 18398 payload 824 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 1 -- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39340a8970 msgr2=0x7f393413df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39340a8970 0x7f393413df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 1 -- 172.21.15.129:0/1394425300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393413f580 msgr2=0x7f39341da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393413f580 0x7f39341da760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 1 monclient: found mon.noname-c 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-08T00:18:08.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.793+0000 7f39337fe640 1 -- 172.21.15.129:0/1394425300 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c009a70 con 0x7f39340a9bd0 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.795+0000 7f39337fe640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39340a9bd0 0x7f39341da220 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7f392400c400 tx=0x7f392400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.795+0000 7f3930ff9640 10 client.?.objecter ms_handle_connect 0x7f39340a9bd0 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.795+0000 7f3930ff9640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f3930ff9640 1 -- 172.21.15.129:0/1394425300 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240052d0 con 0x7f39340a9bd0 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f3930ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f3930ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f3930ff9640 10 monclient: dump: 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.801 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f3930ff9640 10 monclient: _finish_auth 0 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f3930ff9640 10 monclient: _check_auth_tickets 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f3930ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 5 monclient: authenticate success, global_id 18398 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 10 monclient: _renew_subs 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39341d71f0 con 0x7f39340a9bd0 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 10 client.18398.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 10 monclient: _renew_subs 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39341d7700 con 0x7f39340a9bd0 2023-12-08T00:18:08.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.798+0000 7f393a3b6640 1 librados: init done 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f393a3b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f393a3b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f393a3b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39341db0b0 con 0x7f39340a9bd0 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f3930ff9640 1 -- 172.21.15.129:0/1394425300 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3924007e50 con 0x7f39340a9bd0 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f3930ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f3930ff9640 1 -- 172.21.15.129:0/1394425300 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240035f0 con 0x7f39340a9bd0 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f3930ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f3930ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f39327fc640 4 set_mon_vals no callback set 2023-12-08T00:18:08.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:08.799+0000 7f3930ff9640 10 monclient: dump: 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:08.804 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.043 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.042+0000 7f3930ff9640 1 -- 172.21.15.129:0/1394425300 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3924003750 con 0x7f39340a9bd0 2023-12-08T00:18:09.044 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.042+0000 7f3930ff9640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f391406c850 0x7f391406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.044 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.043+0000 7f3933fff640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f391406c850 0x7f391406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.044 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.043+0000 7f3933fff640 10 monclient: get_auth_request con 0x7f391406c850 auth_method 0 2023-12-08T00:18:09.044 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.043+0000 7f3933fff640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f391406c850 0x7f391406ed10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f391c007f70 tx=0x7f391c007f00 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.045 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.043+0000 7f3930ff9640 10 client.18398.objecter ms_handle_connect 0x7f391406c850 2023-12-08T00:18:09.057 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 1 -- 172.21.15.129:0/1394425300 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f3924057f60 con 0x7f39340a9bd0 2023-12-08T00:18:09.057 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 10 client.18398.objecter ms_dispatch 0x7f3934000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:09.057 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 3 client.18398.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:09.057 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 3 client.18398.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:09.057 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 20 client.18398.objecter dump_active .. 0 homeless 2023-12-08T00:18:09.057 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 1 -- 172.21.15.129:0/1394425300 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39240542a0 con 0x7f39340a9bd0 2023-12-08T00:18:09.057 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:09.058 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.055+0000 7f3930ff9640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:09.224 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.223+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f39340a8d50 con 0x7f391406c850 2023-12-08T00:18:09.225 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.223+0000 7f3930ff9640 1 -- 172.21.15.129:0/1394425300 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f391c004c70 con 0x7f391406c850 2023-12-08T00:18:09.225 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 3 to deep-scrub 2023-12-08T00:18:09.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 10 librados: watch_flush enter 2023-12-08T00:18:09.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 10 librados: watch_flush exit 2023-12-08T00:18:09.227 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 20 client.18398.objecter shutdown clearing up homeless session... 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 10 client.18398.objecter successfully canceled tick 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f391406c850 msgr2=0x7f391406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f391406c850 0x7f391406ed10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f391c007f70 tx=0x7f391c007f00 comp rx=0 tx=0).stop 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 10 monclient: shutdown 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39340a9bd0 msgr2=0x7f39341da220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39340a9bd0 0x7f39341da220 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7f392400c400 tx=0x7f392400c8c0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 shutdown_connections 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f391406c850 0x7f391406ed10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393413f580 0x7f39341da760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39340a9bd0 0x7f39341da220 unknown :-1 s=CLOSED pgs=3483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 --2- 172.21.15.129:0/1394425300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39340a8970 0x7f393413df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.225+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 >> 172.21.15.129:0/1394425300 conn(0x7f3934066dd0 msgr2=0x7f39340b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.226+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 shutdown_connections 2023-12-08T00:18:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.226+0000 7f393a3b6640 1 -- 172.21.15.129:0/1394425300 wait complete. 2023-12-08T00:18:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.226+0000 7f393a3b6640 1 librados: shutdown 2023-12-08T00:18:09.239 INFO:tasks.ceph:Scrubbing osd.4 2023-12-08T00:18:09.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:09.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.344+0000 7f51e55c7640 10 monclient: get_monmap_and_config 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.344+0000 7f51e55c7640 10 monclient: build_initial_monmap 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.344+0000 7f51e55c7640 10 monclient: monmap: 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:09.344623+0000 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:09.344623+0000 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:09.347 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 1 Processor -- start 2023-12-08T00:18:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 1 -- start start 2023-12-08T00:18:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient: init 2023-12-08T00:18:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-12-08T00:18:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51e00a8830 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51e0142a40 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f51e013f4c0 0x7f51e014d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient(hunting): picked mon.noname-c con 0x7f51e013f4c0 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e00aa7f0 con 0x7f51e00a8830 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e00aaad0 con 0x7f51e0142a40 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e0141f10 con 0x7f51e013f4c0 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.345+0000 7f51e55c7640 10 monclient(hunting): authenticate will time out at 3289.532751s 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.346+0000 7f51df7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f51e013f4c0 0x7f51e014d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.346+0000 7f51df7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f51e013f4c0 0x7f51e014d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:48062/0 (socket says 172.21.15.129:48062) 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.346+0000 7f51df7fe640 1 -- 172.21.15.129:0/3383779098 learned_addr learned my addr 172.21.15.129:0/3383779098 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:09.349 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.346+0000 7f51df7fe640 10 monclient(hunting): get_auth_request con 0x7f51e013f4c0 auth_method 0 2023-12-08T00:18:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.346+0000 7f51df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.346+0000 7f51df7fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.346+0000 7f51df7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.347+0000 7f51df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.347+0000 7f51df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.347+0000 7f51df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 10 monclient(hunting): handle_auth_done global_id 20266 payload 274 2023-12-08T00:18:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 1 -- 172.21.15.129:0/3383779098 >> v1:172.21.15.184:6789/0 conn(0x7f51e0142a40 legacy=0x7f51e013edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 1 -- 172.21.15.129:0/3383779098 >> v1:172.21.15.129:6789/0 conn(0x7f51e00a8830 legacy=0x7f51e0142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 1 monclient: found mon.noname-c 2023-12-08T00:18:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.349+0000 7f51df7fe640 1 -- 172.21.15.129:0/3383779098 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e00aadb0 con 0x7f51e013f4c0 2023-12-08T00:18:09.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.352+0000 7f51df7fe640 1 --2- 172.21.15.129:0/3383779098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e013f4c0 0x7f51e014d730 secure :-1 s=READY pgs=4869 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009e80 tx=0x7f51d802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a69f922bd2104fe server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 1 -- 172.21.15.129:0/3383779098 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d802caf0 con 0x7f51e013f4c0 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: dump: 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.359 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: _finish_auth 0 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: _check_auth_tickets 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 1 -- 172.21.15.129:0/3383779098 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d802cc50 con 0x7f51e013f4c0 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 1 -- 172.21.15.129:0/3383779098 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d80343f0 con 0x7f51e013f4c0 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51e55c7640 5 monclient: authenticate success, global_id 20266 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51e55c7640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51e55c7640 4 set_mon_vals no callback set 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.357+0000 7f51ddffb640 10 monclient: dump: 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.361 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 10 monclient: shutdown 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 1 -- 172.21.15.129:0/3383779098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e013f4c0 msgr2=0x7f51e014d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 1 --2- 172.21.15.129:0/3383779098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e013f4c0 0x7f51e014d730 secure :-1 s=READY pgs=4869 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009e80 tx=0x7f51d802bf50 comp rx=0 tx=0).stop 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 1 -- 172.21.15.129:0/3383779098 shutdown_connections 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 1 --2- 172.21.15.129:0/3383779098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e013f4c0 0x7f51e014d730 unknown :-1 s=CLOSED pgs=4869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 1 -- 172.21.15.129:0/3383779098 >> 172.21.15.129:0/3383779098 conn(0x7f51e0066c60 msgr2=0x7f51e00b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 1 -- 172.21.15.129:0/3383779098 shutdown_connections 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 1 -- 172.21.15.129:0/3383779098 wait complete. 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 10 monclient: build_initial_monmap 2023-12-08T00:18:09.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.358+0000 7f51e55c7640 10 monclient: monmap: 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 1 librados: starting msgr at 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 1 librados: starting objecter 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 1 Processor -- start 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 1 -- start start 2023-12-08T00:18:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 1 librados: setting wanted keys 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 1 librados: calling monclient init 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 10 monclient: init 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.359+0000 7f51e55c7640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e00a8830 0x7f51e0135780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): picked mon.noname-b con 0x7f51e00a8830 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51e013f4c0 0x7f51e0145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): picked mon.noname-c con 0x7f51e013f4c0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51e0142a40 0x7f51e01d73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): picked mon.noname-a con 0x7f51e0142a40 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01d4aa0 con 0x7f51e0142a40 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0137400 con 0x7f51e00a8830 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01d6ea0 con 0x7f51e013f4c0 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51e55c7640 10 monclient(hunting): authenticate will time out at 3289.547056s 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51deffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e00a8830 0x7f51e0135780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51deffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e00a8830 0x7f51e0135780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:48076/0 (socket says 172.21.15.129:48076) 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51deffd640 1 -- 172.21.15.129:0/1055324494 learned_addr learned my addr 172.21.15.129:0/1055324494 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51deffd640 10 monclient(hunting): get_auth_request con 0x7f51e00a8830 auth_method 0 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51deffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.360+0000 7f51deffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.361+0000 7f51deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.361+0000 7f51deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.361+0000 7f51deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 10 monclient(hunting): handle_auth_done global_id 20269 payload 824 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 1 -- 172.21.15.129:0/1055324494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51e013f4c0 msgr2=0x7f51e0145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f51e013f4c0 0x7f51e0145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 1 -- 172.21.15.129:0/1055324494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51e0142a40 msgr2=0x7f51e01d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51e0142a40 0x7f51e01d73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 10 monclient: _finish_hunting 0 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 1 monclient: found mon.noname-b 2023-12-08T00:18:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.364+0000 7f51deffd640 1 -- 172.21.15.129:0/1055324494 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d8009b30 con 0x7f51e00a8830 2023-12-08T00:18:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.366+0000 7f51deffd640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e00a8830 0x7f51e0135780 secure :-1 s=READY pgs=4870 cs=0 l=1 rev1=1 crypto rx=0x7f51d000c410 tx=0x7f51d000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.366+0000 7f51c77fe640 10 client.?.objecter ms_handle_connect 0x7f51e00a8830 2023-12-08T00:18:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.366+0000 7f51c77fe640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00035f0 con 0x7f51e00a8830 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: dump: 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d0003770 con 0x7f51e00a8830 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0003b20 con 0x7f51e00a8830 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 5 monclient: authenticate success, global_id 20269 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 10 monclient: _renew_subs 2023-12-08T00:18:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e0136280 con 0x7f51e00a8830 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51c77fe640 10 monclient: dump: 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51dd7fa640 4 set_mon_vals no callback set 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 10 client.20269.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 10 monclient: _renew_subs 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e01d7b70 con 0x7f51e00a8830 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.369+0000 7f51e55c7640 1 librados: init done 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.370+0000 7f51e55c7640 10 client.20269.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.370+0000 7f51e55c7640 15 client.20269.objecter _session_command_op_assign -1 1 2023-12-08T00:18:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.370+0000 7f51e55c7640 20 client.20269.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.370+0000 7f51e55c7640 10 client.20269.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.370+0000 7f51e55c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.370+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51e0067ba0 con 0x7f51e00a8830 2023-12-08T00:18:09.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.387+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f51d0003c80 con 0x7f51e00a8830 2023-12-08T00:18:09.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.388+0000 7f51c77fe640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f51cc06c820 0x7f51cc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.388+0000 7f51de7fc640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f51cc06c820 0x7f51cc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.388+0000 7f51de7fc640 10 monclient: get_auth_request con 0x7f51cc06c820 auth_method 0 2023-12-08T00:18:09.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.388+0000 7f51c77fe640 10 client.20269.objecter ms_handle_connect 0x7f51cc06c820 2023-12-08T00:18:09.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.388+0000 7f51de7fc640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f51cc06c820 0x7f51cc06ece0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f51d4000f60 tx=0x7f51d40079d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f51d00587d0 con 0x7f51e00a8830 2023-12-08T00:18:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 10 client.20269.objecter ms_dispatch 0x7f51e0000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 3 client.20269.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 3 client.20269.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 10 client.20269.objecter checking command 1 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f51cc074520 0x7f51cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 20 client.20269.objecter _get_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 20 client.20269.objecter put_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 15 client.20269.objecter _session_command_op_remove -1 1 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 20 client.20269.objecter _get_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 20 client.20269.objecter get_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 15 client.20269.objecter _session_command_op_assign 4 1 2023-12-08T00:18:09.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 20 client.20269.objecter put_session s=0x7f51cc073e50 osd=4 4 2023-12-08T00:18:09.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 10 client.20269.objecter _send_command 1 2023-12-08T00:18:09.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 --> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51cc077090 con 0x7f51cc074520 2023-12-08T00:18:09.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51c77fe640 20 client.20269.objecter dump_active .. 0 homeless 2023-12-08T00:18:09.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.391+0000 7f51df7fe640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f51cc074520 0x7f51cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.392+0000 7f51df7fe640 10 monclient: get_auth_request con 0x7f51cc074520 auth_method 0 2023-12-08T00:18:09.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.394+0000 7f51df7fe640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f51cc074520 0x7f51cc0769c0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.394+0000 7f51c77fe640 10 client.20269.objecter ms_handle_connect 0x7f51cc074520 2023-12-08T00:18:09.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.396+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=1047) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51d0054410 con 0x7f51e00a8830 2023-12-08T00:18:09.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.397+0000 7f51c77fe640 10 monclient: handle_get_version_reply finishing 1 version 1047 2023-12-08T00:18:09.419 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.418+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== osd.4 v2:172.21.15.184:6826/917696104 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f51cc077090 con 0x7f51cc074520 2023-12-08T00:18:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.418+0000 7f51c77fe640 10 client.20269.objecter ms_dispatch 0x7f51e0000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.418+0000 7f51c77fe640 10 client.20269.objecter _finish_command 1 = system:0 2023-12-08T00:18:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.418+0000 7f51c77fe640 20 client.20269.objecter put_session s=0x7f51cc073e50 osd=4 4 2023-12-08T00:18:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.418+0000 7f51c77fe640 15 client.20269.objecter _session_command_op_remove 4 1 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 10 client.20269.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 15 client.20269.objecter _session_command_op_assign -1 2 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 20 client.20269.objecter _get_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 20 client.20269.objecter put_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 20 client.20269.objecter _get_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 15 client.20269.objecter _session_command_op_remove -1 2 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 20 client.20269.objecter get_session s=0x7f51cc073e50 osd=4 3 2023-12-08T00:18:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 15 client.20269.objecter _session_command_op_assign 4 2 2023-12-08T00:18:09.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 20 client.20269.objecter put_session s=0x7f51cc073e50 osd=4 4 2023-12-08T00:18:09.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 10 client.20269.objecter _send_command 2 2023-12-08T00:18:09.434 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.431+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 --> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f51e004e840 con 0x7f51cc074520 2023-12-08T00:18:09.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51c77fe640 1 -- 172.21.15.129:0/1055324494 <== osd.4 v2:172.21.15.184:6826/917696104 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f51e004e840 con 0x7f51cc074520 2023-12-08T00:18:09.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51c77fe640 10 client.20269.objecter ms_dispatch 0x7f51e0000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:09.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51c77fe640 10 client.20269.objecter _finish_command 2 = system:0 2023-12-08T00:18:09.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51c77fe640 20 client.20269.objecter put_session s=0x7f51cc073e50 osd=4 4 2023-12-08T00:18:09.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51c77fe640 15 client.20269.objecter _session_command_op_remove 4 2 2023-12-08T00:18:09.446 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:09.446 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:09.446 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:09.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 10 librados: watch_flush enter 2023-12-08T00:18:09.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 10 librados: watch_flush exit 2023-12-08T00:18:09.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 10 client.20269.objecter close_session for osd.4 2023-12-08T00:18:09.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f51cc074520 msgr2=0x7f51cc0769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6826/917696104,v1:172.21.15.184:6827/917696104] conn(0x7f51cc074520 0x7f51cc0769c0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 20 client.20269.objecter put_session s=0x7f51cc073e50 osd=4 1 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 20 client.20269.objecter shutdown clearing up homeless session... 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 10 client.20269.objecter successfully canceled tick 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f51cc06c820 msgr2=0x7f51cc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f51cc06c820 0x7f51cc06ece0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f51d4000f60 tx=0x7f51d40079d0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 10 monclient: shutdown 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e00a8830 msgr2=0x7f51e0135780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51e55c7640 1 --2- 172.21.15.129:0/1055324494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e00a8830 0x7f51e0135780 secure :-1 s=READY pgs=4870 cs=0 l=1 rev1=1 crypto rx=0x7f51d000c410 tx=0x7f51d000c8d0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.444+0000 7f51df7fe640 1 -- 172.21.15.129:0/1055324494 reap_dead start 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.445+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 shutdown_connections 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.445+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 >> 172.21.15.129:0/1055324494 conn(0x7f51e0066c60 msgr2=0x7f51e013eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.445+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 shutdown_connections 2023-12-08T00:18:09.448 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.445+0000 7f51e55c7640 1 -- 172.21.15.129:0/1055324494 wait complete. 2023-12-08T00:18:09.449 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.445+0000 7f51e55c7640 1 librados: shutdown 2023-12-08T00:18:09.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-08T00:18:09.560 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.559+0000 7f4188f90640 10 monclient: get_monmap_and_config 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.559+0000 7f4188f90640 10 monclient: build_initial_monmap 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.559+0000 7f4188f90640 10 monclient: monmap: 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:09.559639+0000 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:09.559639+0000 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:09.561 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.564 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 1 Processor -- start 2023-12-08T00:18:09.564 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 1 -- start start 2023-12-08T00:18:09.564 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient: init 2023-12-08T00:18:09.564 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41840ad900 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f41840adce0 0x7f4184143180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient(hunting): picked mon.noname-c con 0x7f41840adce0 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41841436c0 0x7f418414a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient(hunting): picked mon.noname-b con 0x7f41841436c0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41840a8cb0 con 0x7f41840ad900 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418413d740 con 0x7f41840adce0 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41840ae910 con 0x7f41841436c0 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:09.565 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.560+0000 7f4188f90640 10 monclient(hunting): authenticate will time out at 3289.747772s 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4182d9d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41841436c0 0x7f418414a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4179d9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f41840adce0 0x7f4184143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f418259c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f41840ad900 0x7f4184066280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:57462/0 (socket says 172.21.15.129:57462) 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f418259c640 1 -- 172.21.15.129:0/2136585699 learned_addr learned my addr 172.21.15.129:0/2136585699 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4182d9d640 10 monclient(hunting): get_auth_request con 0x7f41841436c0 auth_method 0 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4182d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4182d9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4182d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4179d9b640 10 monclient(hunting): get_auth_request con 0x7f41840adce0 auth_method 0 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4179d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4179d9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.561+0000 7f4179d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.562+0000 7f4182d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.562+0000 7f4182d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:09.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.562+0000 7f4182d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:09.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.562+0000 7f4179d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:09.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.562+0000 7f4179d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:09.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.562+0000 7f4179d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:09.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 10 monclient(hunting): handle_auth_done global_id 18401 payload 274 2023-12-08T00:18:09.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 1 -- 172.21.15.129:0/2136585699 >> v2:172.21.15.184:3300/0 conn(0x7f41840adce0 msgr2=0x7f4184143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:09.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 1 --2- 172.21.15.129:0/2136585699 >> v2:172.21.15.184:3300/0 conn(0x7f41840adce0 0x7f4184143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 1 -- 172.21.15.129:0/2136585699 >> v1:172.21.15.184:6789/0 conn(0x7f41840ad900 legacy=0x7f4184066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:09.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 1 monclient: found mon.noname-b 2023-12-08T00:18:09.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:09.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-08T00:18:09.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.563+0000 7f4182d9d640 1 -- 172.21.15.129:0/2136585699 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840af320 con 0x7f41841436c0 2023-12-08T00:18:09.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.565+0000 7f4182d9d640 1 --2- 172.21.15.129:0/2136585699 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41841436c0 0x7f418414a510 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f417c009d90 tx=0x7f417c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9928bd8a1f049dd server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 1 -- 172.21.15.129:0/2136585699 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c02cc50 con 0x7f41841436c0 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: dump: 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.570 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: _finish_auth 0 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: _check_auth_tickets 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 1 -- 172.21.15.129:0/2136585699 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c02cdb0 con 0x7f41841436c0 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 1 -- 172.21.15.129:0/2136585699 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c034660 con 0x7f41841436c0 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 5 monclient: authenticate success, global_id 18401 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.571 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 4 set_mon_vals no callback set 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4181d9b640 10 monclient: dump: 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.572 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 10 monclient: shutdown 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 -- 172.21.15.129:0/2136585699 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41841436c0 msgr2=0x7f418414a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 --2- 172.21.15.129:0/2136585699 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41841436c0 0x7f418414a510 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f417c009d90 tx=0x7f417c02bea0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 -- 172.21.15.129:0/2136585699 shutdown_connections 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 --2- 172.21.15.129:0/2136585699 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41841436c0 0x7f418414a510 unknown :-1 s=CLOSED pgs=3484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 --2- 172.21.15.129:0/2136585699 >> v2:172.21.15.184:3300/0 conn(0x7f41840adce0 0x7f4184143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.576 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 -- 172.21.15.129:0/2136585699 >> 172.21.15.129:0/2136585699 conn(0x7f41840b3cc0 msgr2=0x7f41840b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 -- 172.21.15.129:0/2136585699 shutdown_connections 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.569+0000 7f4188f90640 1 -- 172.21.15.129:0/2136585699 wait complete. 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.570+0000 7f4188f90640 10 monclient: build_initial_monmap 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.570+0000 7f4188f90640 10 monclient: monmap: 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:09.577 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.570+0000 7f4188f90640 1 librados: starting msgr at 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.570+0000 7f4188f90640 1 librados: starting objecter 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 Processor -- start 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 -- start start 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 librados: setting wanted keys 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 librados: calling monclient init 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient: init 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41840ad900 0x7f41841367e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): picked mon.noname-b con 0x7f41840ad900 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41840adce0 0x7f418413b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): picked mon.noname-c con 0x7f41840adce0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41841436c0 0x7f418413ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): picked mon.noname-a con 0x7f41841436c0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418413ca20 con 0x7f41841436c0 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418406fdb0 con 0x7f41840ad900 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418413cc20 con 0x7f41840adce0 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.571+0000 7f4188f90640 10 monclient(hunting): authenticate will time out at 3289.758847s 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4179d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41840adce0 0x7f418413b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f418259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41840ad900 0x7f41841367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4182d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41841436c0 0x7f418413ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f418259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41840ad900 0x7f41841367e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:48104/0 (socket says 172.21.15.129:48104) 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f418259c640 1 -- 172.21.15.129:0/2102353506 learned_addr learned my addr 172.21.15.129:0/2102353506 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f418259c640 10 monclient(hunting): get_auth_request con 0x7f41840ad900 auth_method 0 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f418259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f418259c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f418259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4179d9b640 10 monclient(hunting): get_auth_request con 0x7f41840adce0 auth_method 0 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4179d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4179d9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4179d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4182d9d640 10 monclient(hunting): get_auth_request con 0x7f41841436c0 auth_method 0 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4182d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4182d9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.572+0000 7f4182d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f418259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f418259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f418259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f4179d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:09.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f4179d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:09.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f4179d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:09.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f4182d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:09.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f4182d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:09.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.573+0000 7f4182d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 10 monclient(hunting): handle_auth_done global_id 20278 payload 824 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 1 -- 172.21.15.129:0/2102353506 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41840adce0 msgr2=0x7f418413b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41840adce0 0x7f418413b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 1 -- 172.21.15.129:0/2102353506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41841436c0 msgr2=0x7f418413ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41841436c0 0x7f418413ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 10 monclient: _finish_hunting 0 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 1 monclient: found mon.noname-b 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.575+0000 7f418259c640 1 -- 172.21.15.129:0/2102353506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c009a70 con 0x7f41840ad900 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.578+0000 7f418259c640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41840ad900 0x7f41841367e0 secure :-1 s=READY pgs=4872 cs=0 l=1 rev1=1 crypto rx=0x7f417400c4a0 tx=0x7f417400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.578+0000 7f417b7fe640 10 client.?.objecter ms_handle_connect 0x7f41840ad900 2023-12-08T00:18:09.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.578+0000 7f417b7fe640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:09.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 1 -- 172.21.15.129:0/2102353506 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174005ce0 con 0x7f41840ad900 2023-12-08T00:18:09.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:09.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: dump: 2023-12-08T00:18:09.584 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.584 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.584 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:09.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 1 -- 172.21.15.129:0/2102353506 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4174005e40 con 0x7f41840ad900 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 5 monclient: authenticate success, global_id 20278 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 10 monclient: _renew_subs 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418413cf00 con 0x7f41840ad900 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 1 -- 172.21.15.129:0/2102353506 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174007270 con 0x7f41840ad900 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f417b7fe640 10 monclient: dump: 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:09.586 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f418159a640 4 set_mon_vals no callback set 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 10 client.20278.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 10 monclient: _renew_subs 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418413b270 con 0x7f41840ad900 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.581+0000 7f4188f90640 1 librados: init done 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.582+0000 7f4188f90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.582+0000 7f4188f90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.582+0000 7f4188f90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.582+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41840b01d0 con 0x7f41840ad900 2023-12-08T00:18:09.819 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.818+0000 7f417b7fe640 1 -- 172.21.15.129:0/2102353506 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4174007490 con 0x7f41840ad900 2023-12-08T00:18:09.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.818+0000 7f417b7fe640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f416406c870 0x7f416406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:09.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.819+0000 7f4179d9b640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f416406c870 0x7f416406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:09.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.819+0000 7f4179d9b640 10 monclient: get_auth_request con 0x7f416406c870 auth_method 0 2023-12-08T00:18:09.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.819+0000 7f417b7fe640 10 client.20278.objecter ms_handle_connect 0x7f416406c870 2023-12-08T00:18:09.820 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.819+0000 7f4179d9b640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f416406c870 0x7f416406ed30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f41841ccf50 tx=0x7f416c009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.822+0000 7f417b7fe640 1 -- 172.21.15.129:0/2102353506 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f4174057e10 con 0x7f41840ad900 2023-12-08T00:18:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.822+0000 7f417b7fe640 10 client.20278.objecter ms_dispatch 0x7f4184000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.822+0000 7f417b7fe640 3 client.20278.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.822+0000 7f417b7fe640 3 client.20278.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.822+0000 7f417b7fe640 20 client.20278.objecter dump_active .. 0 homeless 2023-12-08T00:18:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.831+0000 7f417b7fe640 1 -- 172.21.15.129:0/2102353506 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4174054150 con 0x7f41840ad900 2023-12-08T00:18:09.833 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.831+0000 7f417b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:09.833 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:09.831+0000 7f417b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:10.035 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.034+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f41840afe90 con 0x7f416406c870 2023-12-08T00:18:10.036 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.034+0000 7f417b7fe640 1 -- 172.21.15.129:0/2102353506 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f416c011a30 con 0x7f416406c870 2023-12-08T00:18:10.036 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 4 to deep-scrub 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 10 librados: watch_flush enter 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 10 librados: watch_flush exit 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 20 client.20278.objecter shutdown clearing up homeless session... 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 10 client.20278.objecter successfully canceled tick 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f416406c870 msgr2=0x7f416406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f416406c870 0x7f416406ed30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f41841ccf50 tx=0x7f416c009190 comp rx=0 tx=0).stop 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 10 monclient: shutdown 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.036+0000 7f4188f90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41840ad900 msgr2=0x7f41841367e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.039 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41840ad900 0x7f41841367e0 secure :-1 s=READY pgs=4872 cs=0 l=1 rev1=1 crypto rx=0x7f417400c4a0 tx=0x7f417400c960 comp rx=0 tx=0).stop 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 shutdown_connections 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f416406c870 0x7f416406ed30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41841436c0 0x7f418413ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41840adce0 0x7f418413b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 --2- 172.21.15.129:0/2102353506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41840ad900 0x7f41841367e0 unknown :-1 s=CLOSED pgs=4872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 >> 172.21.15.129:0/2102353506 conn(0x7f41840b3cc0 msgr2=0x7f41840b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 shutdown_connections 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 -- 172.21.15.129:0/2102353506 wait complete. 2023-12-08T00:18:10.040 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.037+0000 7f4188f90640 1 librados: shutdown 2023-12-08T00:18:10.049 INFO:tasks.ceph:Scrubbing osd.5 2023-12-08T00:18:10.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.156+0000 7fd36b2e1640 10 monclient: get_monmap_and_config 2023-12-08T00:18:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.156+0000 7fd36b2e1640 10 monclient: build_initial_monmap 2023-12-08T00:18:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.156+0000 7fd36b2e1640 10 monclient: monmap: 2023-12-08T00:18:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:10.156183+0000 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:10.156183+0000 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.156+0000 7fd36b2e1640 1 Processor -- start 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.156+0000 7fd36b2e1640 1 -- start start 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.156+0000 7fd36b2e1640 10 monclient: init 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd3640a8920 0x7fd3640a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient(hunting): picked mon.noname-a con 0x7fd3640a8920 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3640b20e0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd36413f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient(hunting): picked mon.noname-b con 0x7fd3640a9900 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3640a9d40 con 0x7fd3640b20e0 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd364145590 con 0x7fd3640a8920 2023-12-08T00:18:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3641460f0 con 0x7fd3640a9900 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36b2e1640 10 monclient(hunting): authenticate will time out at 3290.344391s 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36887c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fd3640b20e0 0x7fd3641459e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:57464/0 (socket says 172.21.15.129:57464) 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36887c640 1 -- 172.21.15.129:0/1089510103 learned_addr learned my addr 172.21.15.129:0/1089510103 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.157+0000 7fd36987e640 1 --2- 172.21.15.129:0/1089510103 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd36413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.158+0000 7fd36987e640 10 monclient(hunting): get_auth_request con 0x7fd3640a9900 auth_method 0 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.158+0000 7fd36987e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.158+0000 7fd36987e640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.158+0000 7fd36987e640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.158+0000 7fd36987e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.158+0000 7fd36987e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.158+0000 7fd36987e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.159+0000 7fd36907d640 1 --2- 172.21.15.129:0/1089510103 >> v2:172.21.15.129:3300/0 conn(0x7fd3640a8920 0x7fd3640a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36907d640 10 monclient(hunting): get_auth_request con 0x7fd3640a8920 auth_method 0 2023-12-08T00:18:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36907d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36907d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36907d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 10 monclient(hunting): handle_auth_done global_id 18407 payload 274 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 1 -- 172.21.15.129:0/1089510103 >> v2:172.21.15.129:3300/0 conn(0x7fd3640a8920 msgr2=0x7fd3640a8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 1 --2- 172.21.15.129:0/1089510103 >> v2:172.21.15.129:3300/0 conn(0x7fd3640a8920 0x7fd3640a8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 1 -- 172.21.15.129:0/1089510103 >> v1:172.21.15.184:6789/0 conn(0x7fd3640b20e0 legacy=0x7fd3641459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 10 monclient: _finish_hunting 0 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 1 monclient: found mon.noname-b 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.160+0000 7fd36987e640 1 -- 172.21.15.129:0/1089510103 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd364146230 con 0x7fd3640a9900 2023-12-08T00:18:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.162+0000 7fd36987e640 1 --2- 172.21.15.129:0/1089510103 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd36413f100 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7fd3600097a0 tx=0x7fd36002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae8741c1a9c1c6a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 1 -- 172.21.15.129:0/1089510103 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36002e070 con 0x7fd3640a9900 2023-12-08T00:18:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: dump: 2023-12-08T00:18:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: _finish_auth 0 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: _check_auth_tickets 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 1 -- 172.21.15.129:0/1089510103 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3600315a0 con 0x7fd3640a9900 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd36b2e1640 5 monclient: authenticate success, global_id 18407 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd36b2e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd36b2e1640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd36b2e1640 4 set_mon_vals no callback set 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 1 -- 172.21.15.129:0/1089510103 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360031930 con 0x7fd3640a9900 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.166+0000 7fd35bfff640 10 monclient: dump: 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 10 monclient: shutdown 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1089510103 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 msgr2=0x7fd36413f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 --2- 172.21.15.129:0/1089510103 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd36413f100 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7fd3600097a0 tx=0x7fd36002b840 comp rx=0 tx=0).stop 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1089510103 shutdown_connections 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 --2- 172.21.15.129:0/1089510103 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd36413f100 unknown :-1 s=CLOSED pgs=3485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 --2- 172.21.15.129:0/1089510103 >> v2:172.21.15.129:3300/0 conn(0x7fd3640a8920 0x7fd3640a8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1089510103 >> 172.21.15.129:0/1089510103 conn(0x7fd364066ca0 msgr2=0x7fd3640b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1089510103 shutdown_connections 2023-12-08T00:18:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.167+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1089510103 wait complete. 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 10 monclient: build_initial_monmap 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 10 monclient: monmap: 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:10.174 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 1 librados: starting msgr at 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 1 librados: starting objecter 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 1 Processor -- start 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 1 -- start start 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 1 librados: setting wanted keys 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 1 librados: calling monclient init 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.168+0000 7fd36b2e1640 10 monclient: init 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3640a8920 0x7fd364136060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): picked mon.noname-a con 0x7fd3640a8920 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd364139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): picked mon.noname-c con 0x7fd3640a9900 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3640b20e0 0x7fd364136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): picked mon.noname-b con 0x7fd3640b20e0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3641404f0 con 0x7fd3640a8920 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd364140760 con 0x7fd3640b20e0 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd364140a40 con 0x7fd3640a9900 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36b2e1640 10 monclient(hunting): authenticate will time out at 3290.356154s 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36907d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3640a8920 0x7fd364136060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.169+0000 7fd36887c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd364139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3640a8920 0x7fd364136060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58974/0 (socket says 172.21.15.129:58974) 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 1 -- 172.21.15.129:0/1313424493 learned_addr learned my addr 172.21.15.129:0/1313424493 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 10 monclient(hunting): get_auth_request con 0x7fd3640a8920 auth_method 0 2023-12-08T00:18:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36887c640 10 monclient(hunting): get_auth_request con 0x7fd3640a9900 auth_method 0 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36887c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36887c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36887c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36887c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36887c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36887c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.170+0000 7fd36907d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.171+0000 7fd36987e640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3640b20e0 0x7fd364136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.171+0000 7fd36987e640 10 monclient(hunting): get_auth_request con 0x7fd3640b20e0 auth_method 0 2023-12-08T00:18:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.171+0000 7fd36987e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.171+0000 7fd36987e640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.171+0000 7fd36987e640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.172+0000 7fd36987e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.172+0000 7fd36987e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.172+0000 7fd36987e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 10 monclient(hunting): handle_auth_done global_id 15783 payload 824 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 1 -- 172.21.15.129:0/1313424493 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 msgr2=0x7fd364139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd3640a9900 0x7fd364139e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 1 -- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3640b20e0 msgr2=0x7fd364136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3640b20e0 0x7fd364136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 1 monclient: found mon.noname-a 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.173+0000 7fd36907d640 1 -- 172.21.15.129:0/1313424493 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360009400 con 0x7fd3640a8920 2023-12-08T00:18:10.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.180+0000 7fd36907d640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3640a8920 0x7fd364136060 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fd35c007e40 tx=0x7fd35c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.180+0000 7fd359ffb640 10 client.?.objecter ms_handle_connect 0x7fd3640a8920 2023-12-08T00:18:10.181 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.180+0000 7fd359ffb640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:10.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c014620 con 0x7fd3640a8920 2023-12-08T00:18:10.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:10.185 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: dump: 2023-12-08T00:18:10.185 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.185 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: _finish_auth 0 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: _check_auth_tickets 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c00ee90 con 0x7fd3640a8920 2023-12-08T00:18:10.186 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd36b2e1640 5 monclient: authenticate success, global_id 15783 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd36b2e1640 10 monclient: _renew_subs 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd36b2e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd364136660 con 0x7fd3640a8920 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c0149c0 con 0x7fd3640a8920 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd359ffb640 10 monclient: dump: 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.187 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.183+0000 7fd35b7fe640 4 set_mon_vals no callback set 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 10 client.15783.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 10 monclient: _renew_subs 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3641370f0 con 0x7fd3640a8920 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 1 librados: init done 2023-12-08T00:18:10.188 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 10 client.15783.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:10.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 15 client.15783.objecter _session_command_op_assign -1 1 2023-12-08T00:18:10.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 20 client.15783.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:10.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 10 client.15783.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:10.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:10.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.184+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd36414c6f0 con 0x7fd3640a8920 2023-12-08T00:18:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.204+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd35c014b20 con 0x7fd3640a8920 2023-12-08T00:18:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.204+0000 7fd359ffb640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd34806c8a0 0x7fd34806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.205+0000 7fd36887c640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd34806c8a0 0x7fd34806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.205+0000 7fd36887c640 10 monclient: get_auth_request con 0x7fd34806c8a0 auth_method 0 2023-12-08T00:18:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.205+0000 7fd359ffb640 10 client.15783.objecter ms_handle_connect 0x7fd34806c8a0 2023-12-08T00:18:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.205+0000 7fd36887c640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd34806c8a0 0x7fd34806ed60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd3641400d0 tx=0x7fd354007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fd35c0599c0 con 0x7fd3640a8920 2023-12-08T00:18:10.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 10 client.15783.objecter ms_dispatch 0x7fd364000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 3 client.15783.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 3 client.15783.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 10 client.15783.objecter checking command 1 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7fd3480745a0 0x7fd348076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 20 client.15783.objecter _get_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 20 client.15783.objecter put_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 15 client.15783.objecter _session_command_op_remove -1 1 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 20 client.15783.objecter _get_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.211+0000 7fd359ffb640 20 client.15783.objecter get_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd359ffb640 15 client.15783.objecter _session_command_op_assign 5 1 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd359ffb640 20 client.15783.objecter put_session s=0x7fd348073ed0 osd=5 4 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd359ffb640 10 client.15783.objecter _send_command 1 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 --> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd348077110 con 0x7fd3480745a0 2023-12-08T00:18:10.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd359ffb640 20 client.15783.objecter dump_active .. 0 homeless 2023-12-08T00:18:10.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=1047) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd35c055d00 con 0x7fd3640a8920 2023-12-08T00:18:10.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd359ffb640 10 monclient: handle_get_version_reply finishing 1 version 1047 2023-12-08T00:18:10.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.212+0000 7fd36987e640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7fd3480745a0 0x7fd348076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.213+0000 7fd36987e640 10 monclient: get_auth_request con 0x7fd3480745a0 auth_method 0 2023-12-08T00:18:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.216+0000 7fd359ffb640 10 client.15783.objecter ms_handle_connect 0x7fd3480745a0 2023-12-08T00:18:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.216+0000 7fd36987e640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7fd3480745a0 0x7fd348076a40 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.241 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.239+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== osd.5 v2:172.21.15.184:6818/4213359653 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd348077110 con 0x7fd3480745a0 2023-12-08T00:18:10.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.239+0000 7fd359ffb640 10 client.15783.objecter ms_dispatch 0x7fd364000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:10.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.239+0000 7fd359ffb640 10 client.15783.objecter _finish_command 1 = system:0 2023-12-08T00:18:10.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.239+0000 7fd359ffb640 20 client.15783.objecter put_session s=0x7fd348073ed0 osd=5 4 2023-12-08T00:18:10.242 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.239+0000 7fd359ffb640 15 client.15783.objecter _session_command_op_remove 5 1 2023-12-08T00:18:10.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 10 client.15783.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:10.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 15 client.15783.objecter _session_command_op_assign -1 2 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 20 client.15783.objecter _get_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 20 client.15783.objecter put_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 20 client.15783.objecter _get_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 15 client.15783.objecter _session_command_op_remove -1 2 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 20 client.15783.objecter get_session s=0x7fd348073ed0 osd=5 3 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 15 client.15783.objecter _session_command_op_assign 5 2 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 20 client.15783.objecter put_session s=0x7fd348073ed0 osd=5 4 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 10 client.15783.objecter _send_command 2 2023-12-08T00:18:10.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.252+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 --> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd364067ba0 con 0x7fd3480745a0 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd359ffb640 1 -- 172.21.15.129:0/1313424493 <== osd.5 v2:172.21.15.184:6818/4213359653 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd364067ba0 con 0x7fd3480745a0 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd359ffb640 10 client.15783.objecter ms_dispatch 0x7fd364000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd359ffb640 10 client.15783.objecter _finish_command 2 = system:0 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd359ffb640 20 client.15783.objecter put_session s=0x7fd348073ed0 osd=5 4 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd359ffb640 15 client.15783.objecter _session_command_op_remove 5 2 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:10.267 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 10 librados: watch_flush enter 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 10 librados: watch_flush exit 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 10 client.15783.objecter close_session for osd.5 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7fd3480745a0 msgr2=0x7fd348076a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6818/4213359653,v1:172.21.15.184:6819/4213359653] conn(0x7fd3480745a0 0x7fd348076a40 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 20 client.15783.objecter put_session s=0x7fd348073ed0 osd=5 1 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 20 client.15783.objecter shutdown clearing up homeless session... 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.265+0000 7fd36b2e1640 10 client.15783.objecter successfully canceled tick 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd34806c8a0 msgr2=0x7fd34806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fd34806c8a0 0x7fd34806ed60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd3641400d0 tx=0x7fd354007e80 comp rx=0 tx=0).stop 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 10 monclient: shutdown 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3640a8920 msgr2=0x7fd364136060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 --2- 172.21.15.129:0/1313424493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3640a8920 0x7fd364136060 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fd35c007e40 tx=0x7fd35c00e4b0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36987e640 1 -- 172.21.15.129:0/1313424493 reap_dead start 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 shutdown_connections 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 >> 172.21.15.129:0/1313424493 conn(0x7fd364066ca0 msgr2=0x7fd3640b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 shutdown_connections 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 -- 172.21.15.129:0/1313424493 wait complete. 2023-12-08T00:18:10.270 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.266+0000 7fd36b2e1640 1 librados: shutdown 2023-12-08T00:18:10.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-08T00:18:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.377+0000 7f7b839b7640 10 monclient: get_monmap_and_config 2023-12-08T00:18:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.377+0000 7f7b839b7640 10 monclient: build_initial_monmap 2023-12-08T00:18:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.377+0000 7f7b839b7640 10 monclient: monmap: 2023-12-08T00:18:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:10.377433+0000 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:10.377433+0000 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:10.379 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.377+0000 7f7b839b7640 1 Processor -- start 2023-12-08T00:18:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 1 -- start start 2023-12-08T00:18:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient: init 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b7c13d300 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7b7c13d6e0 0x7f7b7c0a8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient(hunting): picked mon.noname-c con 0x7f7b7c13d6e0 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b7c072830 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0a84a0 con 0x7f7b7c13d300 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0aa2f0 con 0x7f7b7c072830 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7c1438d0 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.378+0000 7f7b839b7640 10 monclient(hunting): authenticate will time out at 3290.565683s 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.379+0000 7f7b81753640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7b7c13d300 0x7f7b7c0b0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48060/0 (socket says 172.21.15.129:48060) 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.379+0000 7f7b81753640 1 -- 172.21.15.129:0/3375425959 learned_addr learned my addr 172.21.15.129:0/3375425959 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.379+0000 7f7b80f52640 1 --2- 172.21.15.129:0/3375425959 >> v2:172.21.15.184:3300/0 conn(0x7f7b7c13d6e0 0x7f7b7c0a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.380+0000 7f7b80f52640 10 monclient(hunting): get_auth_request con 0x7f7b7c13d6e0 auth_method 0 2023-12-08T00:18:10.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.380+0000 7f7b80f52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.382 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.380+0000 7f7b80f52640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.380+0000 7f7b80f52640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.380+0000 7f7b80f52640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.380+0000 7f7b80f52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.380+0000 7f7b80f52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.381+0000 7f7b73fff640 1 -- 172.21.15.129:0/3375425959 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243480236 0 0) 0x7f7b7c0a84a0 con 0x7f7b7c13d300 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.381+0000 7f7b73fff640 10 monclient(hunting): _negotiate 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.381+0000 7f7b73fff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.381+0000 7f7b73fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.381+0000 7f7b73fff640 10 monclient(hunting): my global_id is 15786 2023-12-08T00:18:10.383 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.381+0000 7f7b73fff640 1 -- 172.21.15.129:0/3375425959 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b68003520 con 0x7f7b7c13d300 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 10 monclient(hunting): handle_auth_done global_id 20287 payload 274 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 1 -- 172.21.15.129:0/3375425959 >> v1:172.21.15.184:6789/0 conn(0x7f7b7c072830 legacy=0x7f7b7c1431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 1 -- 172.21.15.129:0/3375425959 >> v1:172.21.15.129:6789/0 conn(0x7f7b7c13d300 legacy=0x7f7b7c0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 10 monclient: _finish_hunting 0 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 1 monclient: found mon.noname-c 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:10.384 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.382+0000 7f7b80f52640 1 -- 172.21.15.129:0/3375425959 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c0aa560 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.386+0000 7f7b80f52640 1 --2- 172.21.15.129:0/3375425959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c0a8930 secure :-1 s=READY pgs=4875 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0097d0 tx=0x7f7b6c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98505eb5a7b4ea64 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 1 -- 172.21.15.129:0/3375425959 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c031830 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: dump: 2023-12-08T00:18:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: _finish_auth 0 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: _check_auth_tickets 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 1 -- 172.21.15.129:0/3375425959 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c031990 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b839b7640 5 monclient: authenticate success, global_id 20287 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b839b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b839b7640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 1 -- 172.21.15.129:0/3375425959 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c031d20 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b839b7640 4 set_mon_vals no callback set 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.391+0000 7f7b73fff640 10 monclient: dump: 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.396 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.396 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.396 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 10 monclient: shutdown 2023-12-08T00:18:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 1 -- 172.21.15.129:0/3375425959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 msgr2=0x7f7b7c0a8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3375425959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c0a8930 secure :-1 s=READY pgs=4875 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0097d0 tx=0x7f7b6c02b8d0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 1 -- 172.21.15.129:0/3375425959 shutdown_connections 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3375425959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c0a8930 unknown :-1 s=CLOSED pgs=4875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 1 -- 172.21.15.129:0/3375425959 >> 172.21.15.129:0/3375425959 conn(0x7f7b7c06ac00 msgr2=0x7f7b7c0b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 1 -- 172.21.15.129:0/3375425959 shutdown_connections 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.392+0000 7f7b839b7640 1 -- 172.21.15.129:0/3375425959 wait complete. 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 10 monclient: build_initial_monmap 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 10 monclient: monmap: 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 1 librados: starting msgr at 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 1 librados: starting objecter 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 1 Processor -- start 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 1 -- start start 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 1 librados: setting wanted keys 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 1 librados: calling monclient init 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.393+0000 7f7b839b7640 10 monclient: init 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b7c072830 0x7f7b7c13fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): picked mon.noname-c con 0x7f7b7c072830 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c13d300 0x7f7b7c06f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7c13d300 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): picked mon.noname-b con 0x7f7b7c13d6e0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c06e290 con 0x7f7b7c13d300 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c138bd0 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c138eb0 con 0x7f7b7c072830 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b839b7640 10 monclient(hunting): authenticate will time out at 3290.581152s 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b80f52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c13d300 0x7f7b7c06f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b81f54640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b81753640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b7c072830 0x7f7b7c13fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b80f52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c13d300 0x7f7b7c06f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58990/0 (socket says 172.21.15.129:58990) 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b80f52640 1 -- 172.21.15.129:0/3735807222 learned_addr learned my addr 172.21.15.129:0/3735807222 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b80f52640 10 monclient(hunting): get_auth_request con 0x7f7b7c13d300 auth_method 0 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b80f52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b80f52640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b80f52640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b81f54640 10 monclient(hunting): get_auth_request con 0x7f7b7c13d6e0 auth_method 0 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b81f54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b81f54640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.394+0000 7f7b81f54640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81753640 10 monclient(hunting): get_auth_request con 0x7f7b7c072830 auth_method 0 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81753640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81753640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81753640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81f54640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81f54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81f54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b80f52640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b80f52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b80f52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81753640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81753640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.395+0000 7f7b81753640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.397+0000 7f7b81f54640 10 monclient(hunting): handle_auth_done global_id 20293 payload 824 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 1 -- 172.21.15.129:0/3735807222 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b7c072830 msgr2=0x7f7b7c13fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b7c072830 0x7f7b7c13fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 1 -- 172.21.15.129:0/3735807222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c13d300 msgr2=0x7f7b7c06f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c13d300 0x7f7b7c06f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 10 monclient: _finish_hunting 0 2023-12-08T00:18:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 1 monclient: found mon.noname-b 2023-12-08T00:18:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.398+0000 7f7b81f54640 1 -- 172.21.15.129:0/3735807222 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c009400 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.400+0000 7f7b81f54640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c06e6b0 secure :-1 s=READY pgs=4876 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400c690 tx=0x7f7b7400cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.406 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.400+0000 7f7b71ffb640 10 client.?.objecter ms_handle_connect 0x7f7b7c13d6e0 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.400+0000 7f7b71ffb640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 1 -- 172.21.15.129:0/3735807222 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74003b70 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: dump: 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: _finish_auth 0 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: _check_auth_tickets 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 1 -- 172.21.15.129:0/3735807222 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b74003cd0 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 5 monclient: authenticate success, global_id 20293 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 10 monclient: _renew_subs 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c139620 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 1 -- 172.21.15.129:0/3735807222 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74005310 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b71ffb640 10 monclient: dump: 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b737fe640 4 set_mon_vals no callback set 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 10 client.20293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 10 monclient: _renew_subs 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.403+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c06edd0 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.404+0000 7f7b839b7640 1 librados: init done 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.404+0000 7f7b839b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.404+0000 7f7b839b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.404+0000 7f7b839b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.404+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c06fcd0 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.423+0000 7f7b71ffb640 1 -- 172.21.15.129:0/3735807222 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7b74005470 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.424+0000 7f7b71ffb640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b6006c870 0x7f7b6006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.424+0000 7f7b81753640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b6006c870 0x7f7b6006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.425 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.424+0000 7f7b81753640 10 monclient: get_auth_request con 0x7f7b6006c870 auth_method 0 2023-12-08T00:18:10.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.424+0000 7f7b71ffb640 10 client.20293.objecter ms_handle_connect 0x7f7b6006c870 2023-12-08T00:18:10.426 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.424+0000 7f7b81753640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b6006c870 0x7f7b6006ed30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7b78005880 tx=0x7f7b78009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.427 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.426+0000 7f7b71ffb640 1 -- 172.21.15.129:0/3735807222 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1047..1047 src has 978..1047) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f7b74058920 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.429 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.426+0000 7f7b71ffb640 10 client.20293.objecter ms_dispatch 0x7f7b7c000b90 osd_map(1047..1047 src has 978..1047) v4 2023-12-08T00:18:10.429 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.426+0000 7f7b71ffb640 3 client.20293.objecter handle_osd_map got epochs [1047,1047] > 0 2023-12-08T00:18:10.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.426+0000 7f7b71ffb640 3 client.20293.objecter handle_osd_map decoding full epoch 1047 2023-12-08T00:18:10.430 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.426+0000 7f7b71ffb640 20 client.20293.objecter dump_active .. 0 homeless 2023-12-08T00:18:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.655+0000 7f7b71ffb640 1 -- 172.21.15.129:0/3735807222 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b74054c60 con 0x7f7b7c13d6e0 2023-12-08T00:18:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.655+0000 7f7b71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.655+0000 7f7b71ffb640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:10.838 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.837+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b7c0a91e0 con 0x7f7b6006c870 2023-12-08T00:18:10.839 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.837+0000 7f7b71ffb640 1 -- 172.21.15.129:0/3735807222 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7b78001580 con 0x7f7b6006c870 2023-12-08T00:18:10.839 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 5 to deep-scrub 2023-12-08T00:18:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.839+0000 7f7b839b7640 10 librados: watch_flush enter 2023-12-08T00:18:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.839+0000 7f7b839b7640 10 librados: watch_flush exit 2023-12-08T00:18:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 20 client.20293.objecter shutdown clearing up homeless session... 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 10 client.20293.objecter successfully canceled tick 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b6006c870 msgr2=0x7f7b6006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b6006c870 0x7f7b6006ed30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7b78005880 tx=0x7f7b78009040 comp rx=0 tx=0).stop 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 10 monclient: shutdown 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 msgr2=0x7f7b7c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c06e6b0 secure :-1 s=READY pgs=4876 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400c690 tx=0x7f7b7400cb50 comp rx=0 tx=0).stop 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 shutdown_connections 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f7b6006c870 0x7f7b6006ed30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.842 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b7c13d6e0 0x7f7b7c06e6b0 unknown :-1 s=CLOSED pgs=4876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c13d300 0x7f7b7c06f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 --2- 172.21.15.129:0/3735807222 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b7c072830 0x7f7b7c13fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 >> 172.21.15.129:0/3735807222 conn(0x7f7b7c06ac00 msgr2=0x7f7b7c137ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 shutdown_connections 2023-12-08T00:18:10.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 -- 172.21.15.129:0/3735807222 wait complete. 2023-12-08T00:18:10.846 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.840+0000 7f7b839b7640 1 librados: shutdown 2023-12-08T00:18:10.853 INFO:tasks.ceph:Scrubbing osd.6 2023-12-08T00:18:10.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.957+0000 7fb354dd0640 10 monclient: get_monmap_and_config 2023-12-08T00:18:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.957+0000 7fb354dd0640 10 monclient: build_initial_monmap 2023-12-08T00:18:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.957+0000 7fb354dd0640 10 monclient: monmap: 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:10.958497+0000 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:10.958497+0000 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:10.960 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.957+0000 7fb354dd0640 1 Processor -- start 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.957+0000 7fb354dd0640 1 -- start start 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.957+0000 7fb354dd0640 10 monclient: init 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3500a8920 0x7fb3500a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient(hunting): picked mon.noname-a con 0x7fb3500a8920 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb3500b20e0 0x7fb3500a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient(hunting): picked mon.noname-c con 0x7fb3500b20e0 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3501459e0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35013f810 con 0x7fb3501459e0 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3500a9e40 con 0x7fb3500a8920 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb350145590 con 0x7fb3500b20e0 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.958+0000 7fb354dd0640 10 monclient(hunting): authenticate will time out at 3291.146731s 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.959+0000 7fb34dd9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb3500b20e0 0x7fb3500a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.959+0000 7fb34ed9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fb3501459e0 0x7fb35013f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:57476/0 (socket says 172.21.15.129:57476) 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.959+0000 7fb34ed9d640 1 -- 172.21.15.129:0/4191408872 learned_addr learned my addr 172.21.15.129:0/4191408872 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.959+0000 7fb34dd9b640 10 monclient(hunting): get_auth_request con 0x7fb3500b20e0 auth_method 0 2023-12-08T00:18:10.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.959+0000 7fb34dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.959+0000 7fb34dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.959+0000 7fb34dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.960+0000 7fb34dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.960+0000 7fb34dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.964 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.960+0000 7fb34dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 10 monclient(hunting): handle_auth_done global_id 20296 payload 274 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 1 -- 172.21.15.129:0/4191408872 >> v2:172.21.15.129:3300/0 conn(0x7fb3500a8920 msgr2=0x7fb3500a8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 1 --2- 172.21.15.129:0/4191408872 >> v2:172.21.15.129:3300/0 conn(0x7fb3500a8920 0x7fb3500a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 1 -- 172.21.15.129:0/4191408872 >> v1:172.21.15.184:6789/0 conn(0x7fb3501459e0 legacy=0x7fb35013f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 10 monclient: _finish_hunting 0 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 1 monclient: found mon.noname-c 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:10.965 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.962+0000 7fb34dd9b640 1 -- 172.21.15.129:0/4191408872 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35013f9a0 con 0x7fb3500b20e0 2023-12-08T00:18:10.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.965+0000 7fb34dd9b640 1 --2- 172.21.15.129:0/4191408872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3500b20e0 0x7fb3500a9900 secure :-1 s=READY pgs=4878 cs=0 l=1 rev1=1 crypto rx=0x7fb3500aaa50 tx=0x7fb34402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=719f571d61a503e server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 1 -- 172.21.15.129:0/4191408872 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34402e070 con 0x7fb3500b20e0 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: dump: 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.973 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: _finish_auth 0 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: _check_auth_tickets 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 1 -- 172.21.15.129:0/4191408872 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3440315a0 con 0x7fb3500b20e0 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb354dd0640 5 monclient: authenticate success, global_id 20296 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb354dd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:10.974 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 1 -- 172.21.15.129:0/4191408872 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb344031930 con 0x7fb3500b20e0 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb354dd0640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb354dd0640 4 set_mon_vals no callback set 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.969+0000 7fb34d59a640 10 monclient: dump: 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 10 monclient: shutdown 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 -- 172.21.15.129:0/4191408872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3500b20e0 msgr2=0x7fb3500a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 --2- 172.21.15.129:0/4191408872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3500b20e0 0x7fb3500a9900 secure :-1 s=READY pgs=4878 cs=0 l=1 rev1=1 crypto rx=0x7fb3500aaa50 tx=0x7fb34402b840 comp rx=0 tx=0).stop 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 -- 172.21.15.129:0/4191408872 shutdown_connections 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 --2- 172.21.15.129:0/4191408872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3500b20e0 0x7fb3500a9900 unknown :-1 s=CLOSED pgs=4878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 --2- 172.21.15.129:0/4191408872 >> v2:172.21.15.129:3300/0 conn(0x7fb3500a8920 0x7fb3500a8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 -- 172.21.15.129:0/4191408872 >> 172.21.15.129:0/4191408872 conn(0x7fb350066ca0 msgr2=0x7fb3500b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 -- 172.21.15.129:0/4191408872 shutdown_connections 2023-12-08T00:18:10.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.970+0000 7fb354dd0640 1 -- 172.21.15.129:0/4191408872 wait complete. 2023-12-08T00:18:10.980 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 10 monclient: build_initial_monmap 2023-12-08T00:18:10.980 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 10 monclient: monmap: 2023-12-08T00:18:10.980 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:10.981 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:10.982 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 1 librados: starting msgr at 2023-12-08T00:18:10.982 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 1 librados: starting objecter 2023-12-08T00:18:10.982 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 1 Processor -- start 2023-12-08T00:18:10.982 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 1 -- start start 2023-12-08T00:18:10.982 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 1 librados: setting wanted keys 2023-12-08T00:18:10.982 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 1 librados: calling monclient init 2023-12-08T00:18:10.982 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.971+0000 7fb354dd0640 10 monclient: init 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3500a8920 0x7fb3501420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): picked mon.noname-a con 0x7fb3500a8920 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3500b20e0 0x7fb350134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): picked mon.noname-c con 0x7fb3500b20e0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3501459e0 0x7fb3501363b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): picked mon.noname-b con 0x7fb3501459e0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350136b90 con 0x7fb3500a8920 2023-12-08T00:18:10.983 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350134e90 con 0x7fb3501459e0 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb350136e30 con 0x7fb3500b20e0 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.972+0000 7fb354dd0640 10 monclient(hunting): authenticate will time out at 3291.160504s 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34dd9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3500b20e0 0x7fb350134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3501459e0 0x7fb3501363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34dd9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3500b20e0 0x7fb350134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:58584/0 (socket says 172.21.15.129:58584) 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34dd9b640 1 -- 172.21.15.129:0/287415368 learned_addr learned my addr 172.21.15.129:0/287415368 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34dd9b640 10 monclient(hunting): get_auth_request con 0x7fb3500b20e0 auth_method 0 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 10 monclient(hunting): get_auth_request con 0x7fb3501459e0 auth_method 0 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.973+0000 7fb34ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.974+0000 7fb34dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.974+0000 7fb34dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.974+0000 7fb34dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 10 monclient(hunting): handle_auth_done global_id 20299 payload 824 2023-12-08T00:18:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 1 -- 172.21.15.129:0/287415368 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3500b20e0 msgr2=0x7fb350134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3500b20e0 0x7fb350134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 1 -- 172.21.15.129:0/287415368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3500a8920 msgr2=0x7fb3501420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3500a8920 0x7fb3501420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 1 monclient: found mon.noname-b 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:10.986 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.976+0000 7fb34ed9d640 1 -- 172.21.15.129:0/287415368 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb344009400 con 0x7fb3501459e0 2023-12-08T00:18:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.979+0000 7fb34ed9d640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3501459e0 0x7fb3501363b0 secure :-1 s=READY pgs=4879 cs=0 l=1 rev1=1 crypto rx=0x7fb34000caa0 tx=0x7fb34000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.979+0000 7fb33effd640 10 client.?.objecter ms_handle_connect 0x7fb3501459e0 2023-12-08T00:18:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:10.979+0000 7fb33effd640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:11.058 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000ab40 con 0x7fb3501459e0 2023-12-08T00:18:11.058 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: dump: 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:11.059 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: _finish_auth 0 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34000aca0 con 0x7fb3501459e0 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340012600 con 0x7fb3501459e0 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb33effd640 10 monclient: dump: 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.060 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 5 monclient: authenticate success, global_id 20299 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 10 monclient: _renew_subs 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb34cd99640 4 set_mon_vals no callback set 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb350138d40 con 0x7fb3501459e0 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 10 client.20299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 10 monclient: _renew_subs 2023-12-08T00:18:11.061 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3501372c0 con 0x7fb3501459e0 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.055+0000 7fb354dd0640 1 librados: init done 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.056+0000 7fb354dd0640 10 client.20299.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.056+0000 7fb354dd0640 15 client.20299.objecter _session_command_op_assign -1 1 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.056+0000 7fb354dd0640 20 client.20299.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.056+0000 7fb354dd0640 10 client.20299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.056+0000 7fb354dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:11.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.056+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb35014c6f0 con 0x7fb3501459e0 2023-12-08T00:18:11.087 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.084+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb340007070 con 0x7fb3501459e0 2023-12-08T00:18:11.087 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.084+0000 7fb33effd640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb33006c870 0x7fb33006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.087 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.084+0000 7fb34e59c640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb33006c870 0x7fb33006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.084+0000 7fb34e59c640 10 monclient: get_auth_request con 0x7fb33006c870 auth_method 0 2023-12-08T00:18:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.085+0000 7fb33effd640 10 client.20299.objecter ms_handle_connect 0x7fb33006c870 2023-12-08T00:18:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.085+0000 7fb34e59c640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb33006c870 0x7fb33006ed30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb348009710 tx=0x7fb348006c90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:11.089 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1048..1048 src has 978..1048) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fb340058cd0 con 0x7fb3501459e0 2023-12-08T00:18:11.089 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 10 client.20299.objecter ms_dispatch 0x7fb350000b90 osd_map(1048..1048 src has 978..1048) v4 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 3 client.20299.objecter handle_osd_map got epochs [1048,1048] > 0 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 3 client.20299.objecter handle_osd_map decoding full epoch 1048 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 10 client.20299.objecter checking command 1 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fb330074570 0x7fb330076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 20 client.20299.objecter _get_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 20 client.20299.objecter put_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 15 client.20299.objecter _session_command_op_remove -1 1 2023-12-08T00:18:11.090 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 20 client.20299.objecter _get_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 20 client.20299.objecter get_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 15 client.20299.objecter _session_command_op_assign 6 1 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 20 client.20299.objecter put_session s=0x7fb330073ea0 osd=6 4 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 10 client.20299.objecter _send_command 1 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 --> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3300770e0 con 0x7fb330074570 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.086+0000 7fb33effd640 20 client.20299.objecter dump_active .. 0 homeless 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.087+0000 7fb34dd9b640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fb330074570 0x7fb330076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.087+0000 7fb34dd9b640 10 monclient: get_auth_request con 0x7fb330074570 auth_method 0 2023-12-08T00:18:11.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.090+0000 7fb34dd9b640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fb330074570 0x7fb330076a10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:11.093 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.090+0000 7fb33effd640 10 client.20299.objecter ms_handle_connect 0x7fb330074570 2023-12-08T00:18:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.113+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 <== osd.6 v2:172.21.15.184:6807/105483051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb3300770e0 con 0x7fb330074570 2023-12-08T00:18:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.113+0000 7fb33effd640 10 client.20299.objecter ms_dispatch 0x7fb350000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.113+0000 7fb33effd640 10 client.20299.objecter _finish_command 1 = system:0 2023-12-08T00:18:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.113+0000 7fb33effd640 20 client.20299.objecter put_session s=0x7fb330073ea0 osd=6 4 2023-12-08T00:18:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.113+0000 7fb33effd640 15 client.20299.objecter _session_command_op_remove 6 1 2023-12-08T00:18:11.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 10 client.20299.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:11.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 15 client.20299.objecter _session_command_op_assign -1 2 2023-12-08T00:18:11.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 20 client.20299.objecter _get_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 20 client.20299.objecter put_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 20 client.20299.objecter _get_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 15 client.20299.objecter _session_command_op_remove -1 2 2023-12-08T00:18:11.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 20 client.20299.objecter get_session s=0x7fb330073ea0 osd=6 3 2023-12-08T00:18:11.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 15 client.20299.objecter _session_command_op_assign 6 2 2023-12-08T00:18:11.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 20 client.20299.objecter put_session s=0x7fb330073ea0 osd=6 4 2023-12-08T00:18:11.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 10 client.20299.objecter _send_command 2 2023-12-08T00:18:11.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.126+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 --> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb350136f70 con 0x7fb330074570 2023-12-08T00:18:11.132 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:07.606 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.135 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:07.609 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.142 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.140+0000 7fb33effd640 1 -- 172.21.15.129:0/287415368 <== osd.6 v2:172.21.15.184:6807/105483051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb344031ba0 con 0x7fb330074570 2023-12-08T00:18:11.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.140+0000 7fb33effd640 10 client.20299.objecter ms_dispatch 0x7fb350000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:11.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.140+0000 7fb33effd640 10 client.20299.objecter _finish_command 2 = system:0 2023-12-08T00:18:11.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.140+0000 7fb33effd640 20 client.20299.objecter put_session s=0x7fb330073ea0 osd=6 4 2023-12-08T00:18:11.143 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.140+0000 7fb33effd640 15 client.20299.objecter _session_command_op_remove 6 2 2023-12-08T00:18:11.143 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:11.143 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:11.143 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 10 librados: watch_flush enter 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 10 librados: watch_flush exit 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 10 client.20299.objecter close_session for osd.6 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fb330074570 msgr2=0x7fb330076a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6807/105483051,v1:172.21.15.184:6809/105483051] conn(0x7fb330074570 0x7fb330076a10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 20 client.20299.objecter put_session s=0x7fb330073ea0 osd=6 1 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 20 client.20299.objecter shutdown clearing up homeless session... 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 10 client.20299.objecter successfully canceled tick 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb33006c870 msgr2=0x7fb33006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fb33006c870 0x7fb33006ed30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb348009710 tx=0x7fb348006c90 comp rx=0 tx=0).stop 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 10 monclient: shutdown 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 20 monclient: shutdown canceling and discarding version request 1 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3501459e0 msgr2=0x7fb3501363b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 --2- 172.21.15.129:0/287415368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3501459e0 0x7fb3501363b0 secure :-1 s=READY pgs=4879 cs=0 l=1 rev1=1 crypto rx=0x7fb34000caa0 tx=0x7fb34000cf60 comp rx=0 tx=0).stop 2023-12-08T00:18:11.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb34ed9d640 1 -- 172.21.15.129:0/287415368 reap_dead start 2023-12-08T00:18:11.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 shutdown_connections 2023-12-08T00:18:11.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 >> 172.21.15.129:0/287415368 conn(0x7fb350066ca0 msgr2=0x7fb350073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:11.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 shutdown_connections 2023-12-08T00:18:11.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 -- 172.21.15.129:0/287415368 wait complete. 2023-12-08T00:18:11.146 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.141+0000 7fb354dd0640 1 librados: shutdown 2023-12-08T00:18:11.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-08T00:18:11.160 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:07.634 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.185 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:07.659 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.188 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:07.634 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.189 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:07.663 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.192 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:07.638 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.211 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:07.656 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.215 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:07.688 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:11.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.260+0000 7f98f624b640 10 monclient: get_monmap_and_config 2023-12-08T00:18:11.263 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.260+0000 7f98f624b640 10 monclient: build_initial_monmap 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.260+0000 7f98f624b640 10 monclient: monmap: 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:11.261781+0000 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:11.261781+0000 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.261+0000 7f98f624b640 1 Processor -- start 2023-12-08T00:18:11.264 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.261+0000 7f98f624b640 1 -- start start 2023-12-08T00:18:11.265 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.261+0000 7f98f624b640 10 monclient: init 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f98f00a8950 0x7f98f00a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient(hunting): picked mon.noname-c con 0x7f98f00a8950 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b5610 0x7f98f01425a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient(hunting): picked mon.noname-b con 0x7f98f00b5610 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98f0142ae0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f0141e30 con 0x7f98f0142ae0 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f00b5a30 con 0x7f98f00a8950 2023-12-08T00:18:11.266 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:11.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00aa890 con 0x7f98f00b5610 2023-12-08T00:18:11.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:11.267 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.262+0000 7f98f624b640 10 monclient(hunting): authenticate will time out at 3291.450272s 2023-12-08T00:18:11.269 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.267+0000 7f98eeffd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b5610 0x7f98f01425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:11.279 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.277+0000 7f98eeffd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b5610 0x7f98f01425a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:58588/0 (socket says 172.21.15.129:58588) 2023-12-08T00:18:11.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.277+0000 7f98eeffd640 1 -- 172.21.15.129:0/2025897769 learned_addr learned my addr 172.21.15.129:0/2025897769 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:11.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.277+0000 7f98eeffd640 10 monclient(hunting): get_auth_request con 0x7f98f00b5610 auth_method 0 2023-12-08T00:18:11.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.277+0000 7f98eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:11.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.277+0000 7f98eeffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:11.280 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.277+0000 7f98eeffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:11.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.284+0000 7f98eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:11.286 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.284+0000 7f98eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:11.287 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.284+0000 7f98eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 10 monclient(hunting): handle_auth_done global_id 18419 payload 274 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 1 -- 172.21.15.129:0/2025897769 >> v2:172.21.15.184:3300/0 conn(0x7f98f00a8950 msgr2=0x7f98f00a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 1 --2- 172.21.15.129:0/2025897769 >> v2:172.21.15.184:3300/0 conn(0x7f98f00a8950 0x7f98f00a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 1 -- 172.21.15.129:0/2025897769 >> v1:172.21.15.184:6789/0 conn(0x7f98f0142ae0 legacy=0x7f98f013f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 10 monclient: _finish_hunting 0 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 1 monclient: found mon.noname-b 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-08T00:18:11.289 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.286+0000 7f98eeffd640 1 -- 172.21.15.129:0/2025897769 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f00aab00 con 0x7f98f00b5610 2023-12-08T00:18:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.295+0000 7f98eeffd640 1 --2- 172.21.15.129:0/2025897769 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b5610 0x7f98f01425a0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009870 tx=0x7f98d802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4608d3e2c35786 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 1 -- 172.21.15.129:0/2025897769 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8031640 con 0x7f98f00b5610 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: dump: 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.352 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: _finish_auth 0 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: _check_auth_tickets 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 1 -- 172.21.15.129:0/2025897769 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d80317a0 con 0x7f98f00b5610 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 1 -- 172.21.15.129:0/2025897769 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8031b30 con 0x7f98f00b5610 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98ee7fc640 10 monclient: dump: 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.353 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:11.354 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:11.354 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:11.354 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98f624b640 5 monclient: authenticate success, global_id 18419 2023-12-08T00:18:11.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98f624b640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:11.354 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.349+0000 7f98f624b640 4 set_mon_vals no callback set 2023-12-08T00:18:11.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 10 monclient: shutdown 2023-12-08T00:18:11.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:11.356 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 -- 172.21.15.129:0/2025897769 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b5610 msgr2=0x7f98f01425a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 --2- 172.21.15.129:0/2025897769 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b5610 0x7f98f01425a0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009870 tx=0x7f98d802b8f0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 -- 172.21.15.129:0/2025897769 shutdown_connections 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 --2- 172.21.15.129:0/2025897769 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b5610 0x7f98f01425a0 unknown :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 --2- 172.21.15.129:0/2025897769 >> v2:172.21.15.184:3300/0 conn(0x7f98f00a8950 0x7f98f00a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 -- 172.21.15.129:0/2025897769 >> 172.21.15.129:0/2025897769 conn(0x7f98f0066cd0 msgr2=0x7f98f00b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 -- 172.21.15.129:0/2025897769 shutdown_connections 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 1 -- 172.21.15.129:0/2025897769 wait complete. 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 10 monclient: build_initial_monmap 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.350+0000 7f98f624b640 10 monclient: monmap: 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.351+0000 7f98f624b640 1 librados: starting msgr at 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.351+0000 7f98f624b640 1 librados: starting objecter 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.351+0000 7f98f624b640 1 Processor -- start 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.351+0000 7f98f624b640 1 -- start start 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.351+0000 7f98f624b640 1 librados: setting wanted keys 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.351+0000 7f98f624b640 1 librados: calling monclient init 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.351+0000 7f98f624b640 10 monclient: init 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f00a8950 0x7f98f00b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): picked mon.noname-b con 0x7f98f00a8950 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98f00b5610 0x7f98f0132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): picked mon.noname-a con 0x7f98f00b5610 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b38d0 0x7f98f0133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): picked mon.noname-c con 0x7f98f00b38d0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f0143700 con 0x7f98f00b5610 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f01daf10 con 0x7f98f00a8950 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f0143030 con 0x7f98f00b38d0 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:11.359 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.352+0000 7f98f624b640 10 monclient(hunting): authenticate will time out at 3291.540112s 2023-12-08T00:18:11.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.355+0000 7f98eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98f00b5610 0x7f98f0132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:11.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.355+0000 7f98effff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b38d0 0x7f98f0133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:11.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.358+0000 7f98eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98f00b5610 0x7f98f0132b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59014/0 (socket says 172.21.15.129:59014) 2023-12-08T00:18:11.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.358+0000 7f98eeffd640 1 -- 172.21.15.129:0/1982966601 learned_addr learned my addr 172.21.15.129:0/1982966601 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:11.360 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.358+0000 7f98eeffd640 10 monclient(hunting): get_auth_request con 0x7f98f00b5610 auth_method 0 2023-12-08T00:18:11.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.358+0000 7f98eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:11.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.358+0000 7f98eeffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:11.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.358+0000 7f98eeffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:11.361 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.359+0000 7f98ef7fe640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f00a8950 0x7f98f00b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:11.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.360+0000 7f98effff640 10 monclient(hunting): get_auth_request con 0x7f98f00b38d0 auth_method 0 2023-12-08T00:18:11.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.360+0000 7f98effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:11.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.360+0000 7f98effff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:11.362 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.360+0000 7f98effff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:11.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:11.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:11.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:11.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:11.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:11.363 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:11.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98ef7fe640 10 monclient(hunting): get_auth_request con 0x7f98f00a8950 auth_method 0 2023-12-08T00:18:11.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:11.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98ef7fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:11.364 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.361+0000 7f98ef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:11.366 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 10 monclient(hunting): handle_auth_done global_id 18422 payload 824 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 1 -- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f00a8950 msgr2=0x7f98f00b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f00a8950 0x7f98f00b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 1 -- 172.21.15.129:0/1982966601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98f00b5610 msgr2=0x7f98f0132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98f00b5610 0x7f98f0132b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 1 monclient: found mon.noname-c 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-08T00:18:11.367 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.364+0000 7f98effff640 1 -- 172.21.15.129:0/1982966601 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d80094d0 con 0x7f98f00b38d0 2023-12-08T00:18:11.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.372+0000 7f98effff640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b38d0 0x7f98f0133040 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f98e800b410 tx=0x7f98e800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:11.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.372+0000 7f98e7fff640 10 client.?.objecter ms_handle_connect 0x7f98f00b38d0 2023-12-08T00:18:11.374 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.372+0000 7f98e7fff640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:11.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 1 -- 172.21.15.129:0/1982966601 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8011830 con 0x7f98f00b38d0 2023-12-08T00:18:11.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:11.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:11.394 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: dump: 2023-12-08T00:18:11.394 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:11.394 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:11.394 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: _finish_auth 0 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: _check_auth_tickets 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 1 -- 172.21.15.129:0/1982966601 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e80119b0 con 0x7f98f00b38d0 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 1 -- 172.21.15.129:0/1982966601 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8011d60 con 0x7f98f00b38d0 2023-12-08T00:18:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98e7fff640 10 monclient: dump: 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 5 monclient: authenticate success, global_id 18422 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 10 monclient: _renew_subs 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98edffb640 4 set_mon_vals no callback set 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f00b25f0 con 0x7f98f00b38d0 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 10 client.18422.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 10 monclient: _renew_subs 2023-12-08T00:18:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:11.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f013a900 con 0x7f98f00b38d0 2023-12-08T00:18:11.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.391+0000 7f98f624b640 1 librados: init done 2023-12-08T00:18:11.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.392+0000 7f98f624b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:11.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.392+0000 7f98f624b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:11.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.392+0000 7f98f624b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:11.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.392+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f00b72b0 con 0x7f98f00b38d0 2023-12-08T00:18:11.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.434+0000 7f98e7fff640 1 -- 172.21.15.129:0/1982966601 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f98e801a400 con 0x7f98f00b38d0 2023-12-08T00:18:11.437 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.435+0000 7f98e7fff640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f98d006c8a0 0x7f98d006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:11.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.435+0000 7f98ef7fe640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f98d006c8a0 0x7f98d006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:11.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.435+0000 7f98ef7fe640 10 monclient: get_auth_request con 0x7f98d006c8a0 auth_method 0 2023-12-08T00:18:11.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.436+0000 7f98e7fff640 10 client.18422.objecter ms_handle_connect 0x7f98d006c8a0 2023-12-08T00:18:11.438 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.436+0000 7f98ef7fe640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f98d006c8a0 0x7f98d006ed60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009f20 tx=0x7f98e00094f0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 1 -- 172.21.15.129:0/1982966601 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(1048..1048 src has 978..1048) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f98e8058fe0 con 0x7f98f00b38d0 2023-12-08T00:18:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 10 client.18422.objecter ms_dispatch 0x7f98f0000b90 osd_map(1048..1048 src has 978..1048) v4 2023-12-08T00:18:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 3 client.18422.objecter handle_osd_map got epochs [1048,1048] > 0 2023-12-08T00:18:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 3 client.18422.objecter handle_osd_map decoding full epoch 1048 2023-12-08T00:18:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 20 client.18422.objecter dump_active .. 0 homeless 2023-12-08T00:18:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 1 -- 172.21.15.129:0/1982966601 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98e8054c20 con 0x7f98f00b38d0 2023-12-08T00:18:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.725+0000 7f98e7fff640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:11.931 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.929+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f98f01db050 con 0x7f98d006c8a0 2023-12-08T00:18:11.932 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.929+0000 7f98e7fff640 1 -- 172.21.15.129:0/1982966601 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f98f01db050 con 0x7f98d006c8a0 2023-12-08T00:18:11.932 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 6 to deep-scrub 2023-12-08T00:18:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 10 librados: watch_flush enter 2023-12-08T00:18:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 10 librados: watch_flush exit 2023-12-08T00:18:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 20 client.18422.objecter shutdown clearing up homeless session... 2023-12-08T00:18:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 10 client.18422.objecter successfully canceled tick 2023-12-08T00:18:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f98d006c8a0 msgr2=0x7f98d006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f98d006c8a0 0x7f98d006ed60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009f20 tx=0x7f98e00094f0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 10 monclient: shutdown 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b38d0 msgr2=0x7f98f0133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.931+0000 7f98f624b640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b38d0 0x7f98f0133040 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f98e800b410 tx=0x7f98e800b8d0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 shutdown_connections 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f98d006c8a0 0x7f98d006ed60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98f00b38d0 0x7f98f0133040 unknown :-1 s=CLOSED pgs=3487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98f00b5610 0x7f98f0132b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 --2- 172.21.15.129:0/1982966601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f00a8950 0x7f98f00b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 >> 172.21.15.129:0/1982966601 conn(0x7f98f0066cd0 msgr2=0x7f98f013cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 shutdown_connections 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 -- 172.21.15.129:0/1982966601 wait complete. 2023-12-08T00:18:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:11.932+0000 7f98f624b640 1 librados: shutdown 2023-12-08T00:18:11.945 INFO:tasks.ceph:Scrubbing osd.7 2023-12-08T00:18:11.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-08T00:18:12.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.089+0000 7efcc446f640 10 monclient: get_monmap_and_config 2023-12-08T00:18:12.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.089+0000 7efcc446f640 10 monclient: build_initial_monmap 2023-12-08T00:18:12.092 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.089+0000 7efcc446f640 10 monclient: monmap: 2023-12-08T00:18:12.092 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:12.092 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.093 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:12.090273+0000 2023-12-08T00:18:12.093 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:12.090273+0000 2023-12-08T00:18:12.093 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:12.093 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.093 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:12.093 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:12.094 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:12.094 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:12.094 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:12.094 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.094 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.089+0000 7efcc446f640 1 Processor -- start 2023-12-08T00:18:12.094 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.089+0000 7efcc446f640 1 -- start start 2023-12-08T00:18:12.095 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.089+0000 7efcc446f640 10 monclient: init 2023-12-08T00:18:12.098 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:12.098 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-12-08T00:18:12.098 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc0a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.098 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc0a8920 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:12.098 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efcbc0b20e0 0x7efcbc0a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.098 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc0b20e0 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:12.098 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcbc1459e0 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc13f810 con 0x7efcbc1459e0 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcbc0a9e40 con 0x7efcbc0b20e0 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc145590 con 0x7efcbc0a8920 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.090+0000 7efcc446f640 10 monclient(hunting): authenticate will time out at 3292.278526s 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc220b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc0a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc1a0a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efcbc0b20e0 0x7efcbc0a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc1a0a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efcbc0b20e0 0x7efcbc0a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59018/0 (socket says 172.21.15.129:59018) 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc1a0a640 1 -- 172.21.15.129:0/2166740067 learned_addr learned my addr 172.21.15.129:0/2166740067 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc1a0a640 10 monclient(hunting): get_auth_request con 0x7efcbc0b20e0 auth_method 0 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc1a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc1a0a640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.099 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc1a0a640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc220b640 10 monclient(hunting): get_auth_request con 0x7efcbc0a8920 auth_method 0 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc220b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc220b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.091+0000 7efcc220b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.092+0000 7efcc1a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.092+0000 7efcc1a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.092+0000 7efcc1a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.092+0000 7efcc220b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.092+0000 7efcc220b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.092+0000 7efcc220b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.093+0000 7efcc220b640 10 monclient(hunting): handle_auth_done global_id 18425 payload 274 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.093+0000 7efcc220b640 1 -- 172.21.15.129:0/2166740067 >> v2:172.21.15.129:3300/0 conn(0x7efcbc0b20e0 msgr2=0x7efcbc0a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.093+0000 7efcc220b640 1 --2- 172.21.15.129:0/2166740067 >> v2:172.21.15.129:3300/0 conn(0x7efcbc0b20e0 0x7efcbc0a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.093+0000 7efcc220b640 1 -- 172.21.15.129:0/2166740067 >> v1:172.21.15.129:6789/0 conn(0x7efcbc1459e0 legacy=0x7efcbc13f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:12.100 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.094+0000 7efcc220b640 10 monclient: _finish_hunting 0 2023-12-08T00:18:12.101 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.094+0000 7efcc220b640 1 monclient: found mon.noname-b 2023-12-08T00:18:12.101 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.094+0000 7efcc220b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:12.101 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.094+0000 7efcc220b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.129:3301/0 2023-12-08T00:18:12.101 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.094+0000 7efcc220b640 1 -- 172.21.15.129:0/2166740067 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc13f9a0 con 0x7efcbc0a8920 2023-12-08T00:18:12.101 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.096+0000 7efcc220b640 1 --2- 172.21.15.129:0/2166740067 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc0a8d00 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009d90 tx=0x7efcb402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3fac93ebb7bdcf5 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 1 -- 172.21.15.129:0/2166740067 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb402cb30 con 0x7efcbc0a8920 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: dump: 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.103 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.103 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.103 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.103 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.104 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: _finish_auth 0 2023-12-08T00:18:12.104 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: _check_auth_tickets 2023-12-08T00:18:12.104 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 1 -- 172.21.15.129:0/2166740067 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb402cc90 con 0x7efcbc0a8920 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc446f640 5 monclient: authenticate success, global_id 18425 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc446f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc446f640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 1 -- 172.21.15.129:0/2166740067 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4034440 con 0x7efcbc0a8920 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc446f640 4 set_mon_vals no callback set 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.099+0000 7efcc1209640 10 monclient: dump: 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.106 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 10 monclient: shutdown 2023-12-08T00:18:12.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:12.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 -- 172.21.15.129:0/2166740067 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 msgr2=0x7efcbc0a8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 --2- 172.21.15.129:0/2166740067 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc0a8d00 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009d90 tx=0x7efcb402bd80 comp rx=0 tx=0).stop 2023-12-08T00:18:12.109 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 -- 172.21.15.129:0/2166740067 shutdown_connections 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 --2- 172.21.15.129:0/2166740067 >> v2:172.21.15.129:3300/0 conn(0x7efcbc0b20e0 0x7efcbc0a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 --2- 172.21.15.129:0/2166740067 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc0a8d00 unknown :-1 s=CLOSED pgs=3488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 -- 172.21.15.129:0/2166740067 >> 172.21.15.129:0/2166740067 conn(0x7efcbc066ca0 msgr2=0x7efcbc0b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 -- 172.21.15.129:0/2166740067 shutdown_connections 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.100+0000 7efcc446f640 1 -- 172.21.15.129:0/2166740067 wait complete. 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 10 monclient: build_initial_monmap 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 10 monclient: monmap: 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:12.110 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 1 librados: starting msgr at 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 1 librados: starting objecter 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 1 Processor -- start 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 1 -- start start 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 1 librados: setting wanted keys 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 1 librados: calling monclient init 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.101+0000 7efcc446f640 10 monclient: init 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc0a8920 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcbc0b20e0 0x7efcbc1374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc0b20e0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcbc1459e0 0x7efcbc1392a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc1459e0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc137030 con 0x7efcbc0b20e0 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc138930 con 0x7efcbc1459e0 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc136d30 con 0x7efcbc0a8920 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc446f640 10 monclient(hunting): authenticate will time out at 3292.290477s 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.102+0000 7efcc220b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc1a0a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcbc0b20e0 0x7efcbc1374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc2a0c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcbc1459e0 0x7efcbc1392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc136750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:58624/0 (socket says 172.21.15.129:58624) 2023-12-08T00:18:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 1 -- 172.21.15.129:0/2371558650 learned_addr learned my addr 172.21.15.129:0/2371558650 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 10 monclient(hunting): get_auth_request con 0x7efcbc0a8920 auth_method 0 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc1a0a640 10 monclient(hunting): get_auth_request con 0x7efcbc0b20e0 auth_method 0 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc1a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc1a0a640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc1a0a640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc2a0c640 10 monclient(hunting): get_auth_request con 0x7efcbc1459e0 auth_method 0 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc2a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc2a0c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc2a0c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc220b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc1a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.103+0000 7efcc1a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.104+0000 7efcc1a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.104+0000 7efcc2a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.104+0000 7efcc2a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.104+0000 7efcc2a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 10 monclient(hunting): handle_auth_done global_id 18428 payload 824 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 1 -- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcbc1459e0 msgr2=0x7efcbc1392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcbc1459e0 0x7efcbc1392a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 1 -- 172.21.15.129:0/2371558650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcbc0b20e0 msgr2=0x7efcbc1374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcbc0b20e0 0x7efcbc1374c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 10 monclient: _finish_hunting 0 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 1 monclient: found mon.noname-c 2023-12-08T00:18:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-08T00:18:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.106+0000 7efcc220b640 1 -- 172.21.15.129:0/2371558650 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb4009a70 con 0x7efcbc0a8920 2023-12-08T00:18:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.108+0000 7efcc220b640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc136750 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7efcb4037040 tx=0x7efcb4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.108+0000 7efcb2ffd640 10 client.?.objecter ms_handle_connect 0x7efcbc0a8920 2023-12-08T00:18:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.108+0000 7efcb2ffd640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb403e410 con 0x7efcbc0a8920 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: dump: 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: _finish_auth 0 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb403e570 con 0x7efcbc0a8920 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 5 monclient: authenticate success, global_id 18428 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 10 monclient: _renew_subs 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc138ba0 con 0x7efcbc0a8920 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb403e920 con 0x7efcbc0a8920 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb2ffd640 10 monclient: dump: 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcb3fff640 4 set_mon_vals no callback set 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 10 client.18428.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 10 monclient: _renew_subs 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc142470 con 0x7efcbc0a8920 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.112+0000 7efcc446f640 1 librados: init done 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.113+0000 7efcc446f640 10 client.18428.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.113+0000 7efcc446f640 15 client.18428.objecter _session_command_op_assign -1 1 2023-12-08T00:18:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.113+0000 7efcc446f640 20 client.18428.objecter _get_session osd=-1 returning homeless 2023-12-08T00:18:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.113+0000 7efcc446f640 10 client.18428.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.113+0000 7efcc446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.113+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efcbc14c6f0 con 0x7efcbc0a8920 2023-12-08T00:18:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.131+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efcb403b050 con 0x7efcbc0a8920 2023-12-08T00:18:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.131+0000 7efcb2ffd640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efca006c870 0x7efca006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.131+0000 7efcc1a0a640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efca006c870 0x7efca006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.132+0000 7efcc1a0a640 10 monclient: get_auth_request con 0x7efca006c870 auth_method 0 2023-12-08T00:18:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.132+0000 7efcc1a0a640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efca006c870 0x7efca006ed30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efcbc13fcf0 tx=0x7efcac009240 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.132+0000 7efcb2ffd640 10 client.18428.objecter ms_handle_connect 0x7efca006c870 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(1048..1048 src has 978..1048) v4 ==== 6421+0+0 (secure 0 0 0) 0x7efcb407b2c0 con 0x7efcbc0a8920 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 10 client.18428.objecter ms_dispatch 0x7efcbc000b90 osd_map(1048..1048 src has 978..1048) v4 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 3 client.18428.objecter handle_osd_map got epochs [1048,1048] > 0 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 3 client.18428.objecter handle_osd_map decoding full epoch 1048 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 10 client.18428.objecter checking command 1 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efca0074570 0x7efca0076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 20 client.18428.objecter _get_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 20 client.18428.objecter put_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 15 client.18428.objecter _session_command_op_remove -1 1 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 20 client.18428.objecter _get_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 20 client.18428.objecter get_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 15 client.18428.objecter _session_command_op_assign 7 1 2023-12-08T00:18:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 20 client.18428.objecter put_session s=0x7efca0073ea0 osd=7 4 2023-12-08T00:18:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 10 client.18428.objecter _send_command 1 2023-12-08T00:18:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 --> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efca00770e0 con 0x7efca0074570 2023-12-08T00:18:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.134+0000 7efcb2ffd640 20 client.18428.objecter dump_active .. 0 homeless 2023-12-08T00:18:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.135+0000 7efcc2a0c640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efca0074570 0x7efca0076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.135+0000 7efcc2a0c640 10 monclient: get_auth_request con 0x7efca0074570 auth_method 0 2023-12-08T00:18:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.136+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=1048) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcb4077600 con 0x7efcbc0a8920 2023-12-08T00:18:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.136+0000 7efcb2ffd640 10 monclient: handle_get_version_reply finishing 1 version 1048 2023-12-08T00:18:12.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.139+0000 7efcc2a0c640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efca0074570 0x7efca0076a10 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.139+0000 7efcb2ffd640 10 client.18428.objecter ms_handle_connect 0x7efca0074570 2023-12-08T00:18:12.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.163+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== osd.7 v2:172.21.15.184:6802/59760990 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7efca00770e0 con 0x7efca0074570 2023-12-08T00:18:12.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.163+0000 7efcb2ffd640 10 client.18428.objecter ms_dispatch 0x7efcbc000b90 command_reply(tid 1: 0 ) v1 2023-12-08T00:18:12.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.163+0000 7efcb2ffd640 10 client.18428.objecter _finish_command 1 = system:0 2023-12-08T00:18:12.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.163+0000 7efcb2ffd640 20 client.18428.objecter put_session s=0x7efca0073ea0 osd=7 4 2023-12-08T00:18:12.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.163+0000 7efcb2ffd640 15 client.18428.objecter _session_command_op_remove 7 1 2023-12-08T00:18:12.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 10 client.18428.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-08T00:18:12.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 15 client.18428.objecter _session_command_op_assign -1 2 2023-12-08T00:18:12.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 20 client.18428.objecter _get_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.178 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 20 client.18428.objecter put_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 20 client.18428.objecter _get_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 15 client.18428.objecter _session_command_op_remove -1 2 2023-12-08T00:18:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 20 client.18428.objecter get_session s=0x7efca0073ea0 osd=7 3 2023-12-08T00:18:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 15 client.18428.objecter _session_command_op_assign 7 2 2023-12-08T00:18:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 20 client.18428.objecter put_session s=0x7efca0073ea0 osd=7 4 2023-12-08T00:18:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 10 client.18428.objecter _send_command 2 2023-12-08T00:18:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.176+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 --> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efcbc067ba0 con 0x7efca0074570 2023-12-08T00:18:12.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcb2ffd640 1 -- 172.21.15.129:0/2371558650 <== osd.7 v2:172.21.15.184:6802/59760990 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7efcbc067ba0 con 0x7efca0074570 2023-12-08T00:18:12.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcb2ffd640 10 client.18428.objecter ms_dispatch 0x7efcbc000b90 command_reply(tid 2: 0 ) v1 2023-12-08T00:18:12.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcb2ffd640 10 client.18428.objecter _finish_command 2 = system:0 2023-12-08T00:18:12.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcb2ffd640 20 client.18428.objecter put_session s=0x7efca0073ea0 osd=7 4 2023-12-08T00:18:12.191 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcb2ffd640 15 client.18428.objecter _session_command_op_remove 7 2 2023-12-08T00:18:12.192 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-12-08T00:18:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-08T00:18:12.192 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-12-08T00:18:12.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 10 librados: watch_flush enter 2023-12-08T00:18:12.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 10 librados: watch_flush exit 2023-12-08T00:18:12.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 10 client.18428.objecter close_session for osd.7 2023-12-08T00:18:12.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efca0074570 msgr2=0x7efca0076a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6802/59760990,v1:172.21.15.184:6803/59760990] conn(0x7efca0074570 0x7efca0076a10 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 20 client.18428.objecter put_session s=0x7efca0073ea0 osd=7 1 2023-12-08T00:18:12.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 20 client.18428.objecter shutdown clearing up homeless session... 2023-12-08T00:18:12.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 10 client.18428.objecter successfully canceled tick 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efca006c870 msgr2=0x7efca006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7efca006c870 0x7efca006ed30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efcbc13fcf0 tx=0x7efcac009240 comp rx=0 tx=0).stop 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 10 monclient: shutdown 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 msgr2=0x7efcbc136750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 --2- 172.21.15.129:0/2371558650 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efcbc0a8920 0x7efcbc136750 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7efcb4037040 tx=0x7efcb4004a40 comp rx=0 tx=0).stop 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc2a0c640 1 -- 172.21.15.129:0/2371558650 reap_dead start 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 shutdown_connections 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 >> 172.21.15.129:0/2371558650 conn(0x7efcbc066ca0 msgr2=0x7efcbc0b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 shutdown_connections 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 -- 172.21.15.129:0/2371558650 wait complete. 2023-12-08T00:18:12.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.189+0000 7efcc446f640 1 librados: shutdown 2023-12-08T00:18:12.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-08T00:18:12.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.301+0000 7face0c5d640 10 monclient: get_monmap_and_config 2023-12-08T00:18:12.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.301+0000 7face0c5d640 10 monclient: build_initial_monmap 2023-12-08T00:18:12.303 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.301+0000 7face0c5d640 10 monclient: monmap: 2023-12-08T00:18:12.303 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:12.303 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:12.302249+0000 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:12.302249+0000 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:12.304 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.301+0000 7face0c5d640 1 Processor -- start 2023-12-08T00:18:12.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.301+0000 7face0c5d640 1 -- start start 2023-12-08T00:18:12.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.301+0000 7face0c5d640 10 monclient: init 2023-12-08T00:18:12.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:12.306 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7facdc0a8970 0x7facdc13ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient(hunting): picked mon.noname-a con 0x7facdc0a8970 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facdc0a9bd0 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc13f3b0 0x7facdc1428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient(hunting): picked mon.noname-b con 0x7facdc13f3b0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facdc0aa950 con 0x7facdc0a9bd0 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc142df0 con 0x7facdc0a8970 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc0a90e0 con 0x7facdc13f3b0 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.302+0000 7face0c5d640 10 monclient(hunting): authenticate will time out at 3292.490484s 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facda59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7facdc0a8970 0x7facdc13ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facdad9d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc13f3b0 0x7facdc1428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facd9d9b640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7facdc0a9bd0 0x7facdc1421a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:57492/0 (socket says 172.21.15.129:57492) 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facd9d9b640 1 -- 172.21.15.129:0/1902496171 learned_addr learned my addr 172.21.15.129:0/1902496171 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facda59c640 10 monclient(hunting): get_auth_request con 0x7facdc0a8970 auth_method 0 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facda59c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facda59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facdad9d640 10 monclient(hunting): get_auth_request con 0x7facdc13f3b0 auth_method 0 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facdad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facdad9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.303+0000 7facdad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.308 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.304+0000 7facda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.304+0000 7facda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.304+0000 7facda59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.304+0000 7facdad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.304+0000 7facdad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.309 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.304+0000 7facdad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.310 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facd959a640 1 -- 172.21.15.129:0/1902496171 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473728722 0 0) 0x7facdc0aa950 con 0x7facdc0a9bd0 2023-12-08T00:18:12.310 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facd959a640 10 monclient(hunting): _negotiate 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facd959a640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facd959a640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facd959a640 10 monclient(hunting): my global_id is 20308 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facd959a640 1 -- 172.21.15.129:0/1902496171 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc4003520 con 0x7facdc0a9bd0 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 10 monclient(hunting): handle_auth_done global_id 15801 payload 274 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 1 -- 172.21.15.129:0/1902496171 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc13f3b0 msgr2=0x7facdc1428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 1 --2- 172.21.15.129:0/1902496171 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc13f3b0 0x7facdc1428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 1 -- 172.21.15.129:0/1902496171 >> v1:172.21.15.184:6789/0 conn(0x7facdc0a9bd0 legacy=0x7facdc1421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 10 monclient: _finish_hunting 0 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 1 monclient: found mon.noname-a 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:12.311 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:12.312 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.305+0000 7facda59c640 1 -- 172.21.15.129:0/1902496171 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc13f800 con 0x7facdc0a8970 2023-12-08T00:18:12.312 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.308+0000 7facda59c640 1 --2- 172.21.15.129:0/1902496171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc13ee70 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7facd40097a0 tx=0x7facd402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7578ef74d94eb7 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 1 -- 172.21.15.129:0/1902496171 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd402f050 con 0x7facdc0a8970 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: dump: 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: _finish_auth 0 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: _check_auth_tickets 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 1 -- 172.21.15.129:0/1902496171 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4031700 con 0x7facdc0a8970 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 1 -- 172.21.15.129:0/1902496171 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4031a90 con 0x7facdc0a8970 2023-12-08T00:18:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7face0c5d640 5 monclient: authenticate success, global_id 15801 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7face0c5d640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7face0c5d640 4 set_mon_vals no callback set 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.311+0000 7facd959a640 10 monclient: dump: 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.316 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.317 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.317 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.317 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 10 monclient: shutdown 2023-12-08T00:18:12.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:12.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 -- 172.21.15.129:0/1902496171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 msgr2=0x7facdc13ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 --2- 172.21.15.129:0/1902496171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc13ee70 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7facd40097a0 tx=0x7facd402b7a0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.320 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 -- 172.21.15.129:0/1902496171 shutdown_connections 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 --2- 172.21.15.129:0/1902496171 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc13f3b0 0x7facdc1428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 --2- 172.21.15.129:0/1902496171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc13ee70 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 -- 172.21.15.129:0/1902496171 >> 172.21.15.129:0/1902496171 conn(0x7facdc066dd0 msgr2=0x7facdc0b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 -- 172.21.15.129:0/1902496171 shutdown_connections 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.312+0000 7face0c5d640 1 -- 172.21.15.129:0/1902496171 wait complete. 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.313+0000 7face0c5d640 10 monclient: build_initial_monmap 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.313+0000 7face0c5d640 10 monclient: monmap: 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:12.321 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 librados: starting msgr at 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 librados: starting objecter 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 Processor -- start 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 -- start start 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 librados: setting wanted keys 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 librados: calling monclient init 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient: init 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc0ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): picked mon.noname-a con 0x7facdc0a8970 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:12.322 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc0a9bd0 0x7facdc1d7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): picked mon.noname-c con 0x7facdc0a9bd0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facdc13f3b0 0x7facdc1d77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): picked mon.noname-b con 0x7facdc13f3b0 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc0adc00 con 0x7facdc0a8970 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc0ade70 con 0x7facdc13f3b0 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc1d6d40 con 0x7facdc0a9bd0 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.314+0000 7face0c5d640 10 monclient(hunting): authenticate will time out at 3292.502855s 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facda59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc0ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facdad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facdc13f3b0 0x7facdc1d77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facd9d9b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc0a9bd0 0x7facdc1d7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.323 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facdad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facdc13f3b0 0x7facdc1d77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.129:48202/0 (socket says 172.21.15.129:48202) 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facda59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc0ac5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59044/0 (socket says 172.21.15.129:59044) 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facdad9d640 1 -- 172.21.15.129:0/2900091377 learned_addr learned my addr 172.21.15.129:0/2900091377 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facdad9d640 10 monclient(hunting): get_auth_request con 0x7facdc13f3b0 auth_method 0 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facdad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facdad9d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facdad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facda59c640 10 monclient(hunting): get_auth_request con 0x7facdc0a8970 auth_method 0 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facda59c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.315+0000 7facda59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facd9d9b640 10 monclient(hunting): get_auth_request con 0x7facdc0a9bd0 auth_method 0 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facd9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facd9d9b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facd9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.324 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facdad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facdad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facdad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facda59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facd9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facd9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.325 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.316+0000 7facd9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 10 monclient(hunting): handle_auth_done global_id 20311 payload 824 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 1 -- 172.21.15.129:0/2900091377 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc0a9bd0 msgr2=0x7facdc1d7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc0a9bd0 0x7facdc1d7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 1 -- 172.21.15.129:0/2900091377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 msgr2=0x7facdc0ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc0ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 1 monclient: found mon.noname-b 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.184:3300/0 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.318+0000 7facdad9d640 1 -- 172.21.15.129:0/2900091377 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4009400 con 0x7facdc13f3b0 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.321+0000 7facdad9d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facdc13f3b0 0x7facdc1d77a0 secure :-1 s=READY pgs=4881 cs=0 l=1 rev1=1 crypto rx=0x7faccc00e7f0 tx=0x7faccc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.321+0000 7faccaffd640 10 client.?.objecter ms_handle_connect 0x7facdc13f3b0 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.321+0000 7faccaffd640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7faccaffd640 1 -- 172.21.15.129:0/2900091377 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc009460 con 0x7facdc13f3b0 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7faccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7faccaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7faccaffd640 10 monclient: dump: 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7faccaffd640 10 monclient: _finish_auth 0 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7faccaffd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7faccaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 5 monclient: authenticate success, global_id 20311 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 10 monclient: _renew_subs 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:12.329 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc1393d0 con 0x7facdc13f3b0 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 10 client.20311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 10 monclient: _renew_subs 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc0ab9f0 con 0x7facdc13f3b0 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.323+0000 7face0c5d640 1 librados: init done 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.324+0000 7face0c5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.324+0000 7face0c5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.324+0000 7face0c5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.184:3300/0 2023-12-08T00:18:12.330 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.324+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc04e840 con 0x7facdc13f3b0 2023-12-08T00:18:12.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.530+0000 7faccaffd640 1 -- 172.21.15.129:0/2900091377 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc005990 con 0x7facdc13f3b0 2023-12-08T00:18:12.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.530+0000 7faccaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:12.532 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.530+0000 7facd8d99640 4 set_mon_vals no callback set 2023-12-08T00:18:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.564+0000 7faccaffd640 1 -- 172.21.15.129:0/2900091377 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc005d40 con 0x7facdc13f3b0 2023-12-08T00:18:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.564+0000 7faccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.564+0000 7faccaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.564+0000 7faccaffd640 10 monclient: dump: 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.567 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.565+0000 7faccaffd640 1 -- 172.21.15.129:0/2900091377 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faccc00c730 con 0x7facdc13f3b0 2023-12-08T00:18:12.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.565+0000 7faccaffd640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7facbc06c850 0x7facbc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.565+0000 7facda59c640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7facbc06c850 0x7facbc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.565+0000 7facda59c640 10 monclient: get_auth_request con 0x7facbc06c850 auth_method 0 2023-12-08T00:18:12.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.566+0000 7facda59c640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7facbc06c850 0x7facbc06ed10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7facd4035000 tx=0x7facd4008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.568 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.566+0000 7faccaffd640 10 client.20311.objecter ms_handle_connect 0x7facbc06c850 2023-12-08T00:18:12.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.567+0000 7faccaffd640 1 -- 172.21.15.129:0/2900091377 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(1048..1048 src has 978..1048) v4 ==== 6421+0+0 (secure 0 0 0) 0x7faccc058b70 con 0x7facdc13f3b0 2023-12-08T00:18:12.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.567+0000 7faccaffd640 10 client.20311.objecter ms_dispatch 0x7facdc000b90 osd_map(1048..1048 src has 978..1048) v4 2023-12-08T00:18:12.569 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.567+0000 7faccaffd640 3 client.20311.objecter handle_osd_map got epochs [1048,1048] > 0 2023-12-08T00:18:12.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.567+0000 7faccaffd640 3 client.20311.objecter handle_osd_map decoding full epoch 1048 2023-12-08T00:18:12.570 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.567+0000 7faccaffd640 20 client.20311.objecter dump_active .. 0 homeless 2023-12-08T00:18:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.577+0000 7faccaffd640 1 -- 172.21.15.129:0/2900091377 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faccc054eb0 con 0x7facdc13f3b0 2023-12-08T00:18:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.577+0000 7faccaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.577+0000 7faccaffd640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:12.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.757+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7facdc0a8d50 con 0x7facbc06c850 2023-12-08T00:18:12.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.757+0000 7faccaffd640 1 -- 172.21.15.129:0/2900091377 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7facd4042070 con 0x7facbc06c850 2023-12-08T00:18:12.760 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 7 to deep-scrub 2023-12-08T00:18:12.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 10 librados: watch_flush enter 2023-12-08T00:18:12.762 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 10 librados: watch_flush exit 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 20 client.20311.objecter shutdown clearing up homeless session... 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 10 client.20311.objecter successfully canceled tick 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7facbc06c850 msgr2=0x7facbc06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7facbc06c850 0x7facbc06ed10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7facd4035000 tx=0x7facd4008030 comp rx=0 tx=0).stop 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 10 monclient: shutdown 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facdc13f3b0 msgr2=0x7facdc1d77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facdc13f3b0 0x7facdc1d77a0 secure :-1 s=READY pgs=4881 cs=0 l=1 rev1=1 crypto rx=0x7faccc00e7f0 tx=0x7faccc00ecb0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 shutdown_connections 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7facbc06c850 0x7facbc06ed10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facdc13f3b0 0x7facdc1d77a0 unknown :-1 s=CLOSED pgs=4881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7facdc0a9bd0 0x7facdc1d7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.763 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.759+0000 7face0c5d640 1 --2- 172.21.15.129:0/2900091377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facdc0a8970 0x7facdc0ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.760+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 >> 172.21.15.129:0/2900091377 conn(0x7facdc066dd0 msgr2=0x7facdc0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:12.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.760+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 shutdown_connections 2023-12-08T00:18:12.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.760+0000 7face0c5d640 1 -- 172.21.15.129:0/2900091377 wait complete. 2023-12-08T00:18:12.764 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.760+0000 7face0c5d640 1 librados: shutdown 2023-12-08T00:18:12.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.876+0000 7f92b977d640 10 monclient: get_monmap_and_config 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.876+0000 7f92b977d640 10 monclient: build_initial_monmap 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.876+0000 7f92b977d640 10 monclient: monmap: 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:12.878082+0000 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:12.878082+0000 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:12.880 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:12.880 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:12.880 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:12.880 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:12.880 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.877+0000 7f92b977d640 1 Processor -- start 2023-12-08T00:18:12.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.877+0000 7f92b977d640 1 -- start start 2023-12-08T00:18:12.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.877+0000 7f92b977d640 10 monclient: init 2023-12-08T00:18:12.881 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92b40a8950 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92b40a8d30 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f92b413f820 0x7f92b40b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient(hunting): picked mon.noname-a con 0x7f92b413f820 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b4141e30 con 0x7f92b40a8d30 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b40aa890 con 0x7f92b40a8950 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b4142cb0 con 0x7f92b413f820 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b977d640 10 monclient(hunting): authenticate will time out at 3293.066176s 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b2ffd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f92b40a8950 0x7f92b41425a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.129:55294/0 (socket says 172.21.15.129:55294) 2023-12-08T00:18:12.882 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.878+0000 7f92b2ffd640 1 -- 172.21.15.129:0/1845059694 learned_addr learned my addr 172.21.15.129:0/1845059694 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-12-08T00:18:12.883 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.880+0000 7f92b37fe640 1 --2- 172.21.15.129:0/1845059694 >> v2:172.21.15.129:3300/0 conn(0x7f92b413f820 0x7f92b40b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.884 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.882+0000 7f92b37fe640 10 monclient(hunting): get_auth_request con 0x7f92b413f820 auth_method 0 2023-12-08T00:18:12.884 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.882+0000 7f92b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.884 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.882+0000 7f92b37fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.884 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.882+0000 7f92b37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.885 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.883+0000 7f92b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.885 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.883+0000 7f92b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.885 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.883+0000 7f92b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 10 monclient(hunting): handle_auth_done global_id 15807 payload 274 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 1 -- 172.21.15.129:0/1845059694 >> v1:172.21.15.184:6789/0 conn(0x7f92b40a8950 legacy=0x7f92b41425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 1 -- 172.21.15.129:0/1845059694 >> v1:172.21.15.129:6789/0 conn(0x7f92b40a8d30 legacy=0x7f92b413f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 1 monclient: found mon.noname-a 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:12.890 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.885+0000 7f92b37fe640 1 -- 172.21.15.129:0/1845059694 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b40aab00 con 0x7f92b413f820 2023-12-08T00:18:12.902 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.900+0000 7f92b37fe640 1 --2- 172.21.15.129:0/1845059694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b413f820 0x7f92b40b5610 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009d90 tx=0x7f92a402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49458020bea8fca6 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.961 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 1 -- 172.21.15.129:0/1845059694 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a402cc60 con 0x7f92b413f820 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: dump: 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.962 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: _finish_auth 0 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: _check_auth_tickets 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 1 -- 172.21.15.129:0/1845059694 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a402cdc0 con 0x7f92b413f820 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 1 -- 172.21.15.129:0/1845059694 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4034660 con 0x7f92b413f820 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b977d640 5 monclient: authenticate success, global_id 15807 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b977d640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b977d640 4 set_mon_vals no callback set 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.959+0000 7f92b1ffb640 10 monclient: dump: 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.963 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.964 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.964 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.964 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.964 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.964 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.964 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.964 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 10 monclient: shutdown 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 1 -- 172.21.15.129:0/1845059694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b413f820 msgr2=0x7f92b40b5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 1 --2- 172.21.15.129:0/1845059694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b413f820 0x7f92b40b5610 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009d90 tx=0x7f92a402beb0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 1 -- 172.21.15.129:0/1845059694 shutdown_connections 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 1 --2- 172.21.15.129:0/1845059694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b413f820 0x7f92b40b5610 unknown :-1 s=CLOSED pgs=3278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 1 -- 172.21.15.129:0/1845059694 >> 172.21.15.129:0/1845059694 conn(0x7f92b4066cd0 msgr2=0x7f92b40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 1 -- 172.21.15.129:0/1845059694 shutdown_connections 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 1 -- 172.21.15.129:0/1845059694 wait complete. 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 10 monclient: build_initial_monmap 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.960+0000 7f92b977d640 10 monclient: monmap: 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.961+0000 7f92b977d640 1 librados: starting msgr at 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.961+0000 7f92b977d640 1 librados: starting objecter 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.961+0000 7f92b977d640 1 Processor -- start 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.961+0000 7f92b977d640 1 -- start start 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.961+0000 7f92b977d640 1 librados: setting wanted keys 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.961+0000 7f92b977d640 1 librados: calling monclient init 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.961+0000 7f92b977d640 10 monclient: init 2023-12-08T00:18:12.968 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b40a8950 0x7f92b4138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): picked mon.noname-a con 0x7f92b40a8950 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40a8d30 0x7f92b41458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): picked mon.noname-b con 0x7f92b40a8d30 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b413f820 0x7f92b4145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): picked mon.noname-c con 0x7f92b413f820 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4138060 con 0x7f92b40a8950 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4138340 con 0x7f92b40a8d30 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4138620 con 0x7f92b413f820 2023-12-08T00:18:12.969 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b977d640 10 monclient(hunting): authenticate will time out at 3293.150060s 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b37fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b413f820 0x7f92b4145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b40a8950 0x7f92b4138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b37fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b413f820 0x7f92b4145e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:58172/0 (socket says 172.21.15.129:58172) 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b37fe640 1 -- 172.21.15.129:0/2094794935 learned_addr learned my addr 172.21.15.129:0/2094794935 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b37fe640 10 monclient(hunting): get_auth_request con 0x7f92b413f820 auth_method 0 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b37fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b2ffd640 10 monclient(hunting): get_auth_request con 0x7f92b40a8950 auth_method 0 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.962+0000 7f92b2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:12.970 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.963+0000 7f92b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.971 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.963+0000 7f92b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.971 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.963+0000 7f92b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.971 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.963+0000 7f92b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:12.971 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.963+0000 7f92b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:12.971 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.963+0000 7f92b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:12.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 10 monclient(hunting): handle_auth_done global_id 15810 payload 824 2023-12-08T00:18:12.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 1 -- 172.21.15.129:0/2094794935 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b413f820 msgr2=0x7f92b4145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b413f820 0x7f92b4145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.972 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 1 -- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40a8d30 msgr2=0x7f92b41458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40a8d30 0x7f92b41458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 10 monclient: _finish_hunting 0 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 1 monclient: found mon.noname-a 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.966+0000 7f92b2ffd640 1 -- 172.21.15.129:0/2094794935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a4009a70 con 0x7f92b40a8950 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.968+0000 7f92b2ffd640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b40a8950 0x7f92b4138d20 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00c440 tx=0x7f92ac001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.968+0000 7f929b7fe640 10 client.?.objecter ms_handle_connect 0x7f92b40a8950 2023-12-08T00:18:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.968+0000 7f929b7fe640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 1 -- 172.21.15.129:0/2094794935 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac005ce0 con 0x7f92b40a8950 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: dump: 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: _finish_auth 0 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: _check_auth_tickets 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 1 -- 172.21.15.129:0/2094794935 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac005e40 con 0x7f92b40a8950 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 5 monclient: authenticate success, global_id 15810 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 10 monclient: _renew_subs 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b4138900 con 0x7f92b40a8950 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 1 -- 172.21.15.129:0/2094794935 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac0072d0 con 0x7f92b40a8950 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f929b7fe640 10 monclient: dump: 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b17fa640 4 set_mon_vals no callback set 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 10 client.15810.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 10 monclient: _renew_subs 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:12.977 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b4131690 con 0x7f92b40a8950 2023-12-08T00:18:12.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.972+0000 7f92b977d640 1 librados: init done 2023-12-08T00:18:12.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.973+0000 7f92b977d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:12.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.973+0000 7f92b977d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:12.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.973+0000 7f92b977d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:12.978 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.973+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b40b72b0 con 0x7f92b40a8950 2023-12-08T00:18:12.993 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.991+0000 7f929b7fe640 1 -- 172.21.15.129:0/2094794935 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f92ac0074f0 con 0x7f92b40a8950 2023-12-08T00:18:12.994 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.992+0000 7f929b7fe640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f92a006c870 0x7f92a006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:12.994 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.992+0000 7f92b27fc640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f92a006c870 0x7f92a006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:12.994 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.992+0000 7f92b27fc640 10 monclient: get_auth_request con 0x7f92a006c870 auth_method 0 2023-12-08T00:18:12.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.992+0000 7f92b27fc640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f92a006c870 0x7f92a006ed30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f92b40b1ef0 tx=0x7f92a8007990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:12.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.992+0000 7f929b7fe640 10 client.15810.objecter ms_handle_connect 0x7f92a006c870 2023-12-08T00:18:12.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.996+0000 7f929b7fe640 1 -- 172.21.15.129:0/2094794935 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1048..1048 src has 978..1048) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f92ac057e90 con 0x7f92b40a8950 2023-12-08T00:18:12.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.996+0000 7f929b7fe640 10 client.15810.objecter ms_dispatch 0x7f92b4000b90 osd_map(1048..1048 src has 978..1048) v4 2023-12-08T00:18:12.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.996+0000 7f929b7fe640 3 client.15810.objecter handle_osd_map got epochs [1048,1048] > 0 2023-12-08T00:18:12.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.996+0000 7f929b7fe640 3 client.15810.objecter handle_osd_map decoding full epoch 1048 2023-12-08T00:18:12.998 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:12.996+0000 7f929b7fe640 20 client.15810.objecter dump_active .. 0 homeless 2023-12-08T00:18:13.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.268+0000 7f929b7fe640 1 -- 172.21.15.129:0/2094794935 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92ac0541d0 con 0x7f92b40a8950 2023-12-08T00:18:13.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.268+0000 7f929b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:13.271 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.268+0000 7f929b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:13.436 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.434+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92b4142080 con 0x7f92a006c870 2023-12-08T00:18:13.441 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.439+0000 7f929b7fe640 1 -- 172.21.15.129:0/2094794935 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150742 (secure 0 0 0) 0x7f92a80050d0 con 0x7f92a006c870 2023-12-08T00:18:13.443 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-08T00:18:13.444 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-08T00:18:13.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 10 librados: watch_flush enter 2023-12-08T00:18:13.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 10 librados: watch_flush exit 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 20 client.15810.objecter shutdown clearing up homeless session... 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 10 client.15810.objecter successfully canceled tick 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f92a006c870 msgr2=0x7f92a006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f92a006c870 0x7f92a006ed30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f92b40b1ef0 tx=0x7f92a8007990 comp rx=0 tx=0).stop 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 10 monclient: shutdown 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b40a8950 msgr2=0x7f92b4138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b40a8950 0x7f92b4138d20 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00c440 tx=0x7f92ac001e70 comp rx=0 tx=0).stop 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 shutdown_connections 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f92a006c870 0x7f92a006ed30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b413f820 0x7f92b4145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40a8d30 0x7f92b41458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 --2- 172.21.15.129:0/2094794935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b40a8950 0x7f92b4138d20 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 >> 172.21.15.129:0/2094794935 conn(0x7f92b4066cd0 msgr2=0x7f92b413eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 shutdown_connections 2023-12-08T00:18:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 -- 172.21.15.129:0/2094794935 wait complete. 2023-12-08T00:18:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:13.442+0000 7f92b977d640 1 librados: shutdown 2023-12-08T00:18:13.456 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":2017,"stamp":"2023-12-08T00:18:12.137767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":97,"num_write":134,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":589084,"kb_used_data":49020,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754385636,"statfs":{"total":773094113280,"available":772490891264,"internally_reserved":0,"allocated":50196480,"data_stored":23988127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":112,"apply_latency_ms":112,"commit_latency_ns":112000000,"apply_latency_ns":112000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001547"},"pg_stats":[{"pgid":"15.1f","version":"0'0","reported_seq":1002,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.316516+0000","last_change":"2023-12-07T23:47:23.193178+0000","last_active":"2023-12-08T00:18:02.316516+0000","last_peered":"2023-12-08T00:18:02.316516+0000","last_clean":"2023-12-08T00:18:02.316516+0000","last_became_active":"2023-12-07T23:47:23.187382+0000","last_became_peered":"2023-12-07T23:47:23.187382+0000","last_unstale":"2023-12-08T00:18:02.316516+0000","last_undegraded":"2023-12-08T00:18:02.316516+0000","last_fullsized":"2023-12-08T00:18:02.316516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:46:32.619584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":994,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.602437+0000","last_change":"2023-12-07T23:47:26.349171+0000","last_active":"2023-12-08T00:18:02.602437+0000","last_peered":"2023-12-08T00:18:02.602437+0000","last_clean":"2023-12-08T00:18:02.602437+0000","last_became_active":"2023-12-07T23:47:26.088721+0000","last_became_peered":"2023-12-07T23:47:26.088721+0000","last_unstale":"2023-12-08T00:18:02.602437+0000","last_undegraded":"2023-12-08T00:18:02.602437+0000","last_fullsized":"2023-12-08T00:18:02.602437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:46:10.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1015,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.182052+0000","last_change":"2023-12-07T23:47:18.674241+0000","last_active":"2023-12-08T00:18:07.182052+0000","last_peered":"2023-12-08T00:18:07.182052+0000","last_clean":"2023-12-08T00:18:07.182052+0000","last_became_active":"2023-12-07T23:47:18.658074+0000","last_became_peered":"2023-12-07T23:47:18.658074+0000","last_unstale":"2023-12-08T00:18:07.182052+0000","last_undegraded":"2023-12-08T00:18:07.182052+0000","last_fullsized":"2023-12-08T00:18:07.182052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":990,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.511089+0000","last_change":"2023-12-07T23:47:39.142806+0000","last_active":"2023-12-08T00:18:07.511089+0000","last_peered":"2023-12-08T00:18:07.511089+0000","last_clean":"2023-12-08T00:18:07.511089+0000","last_became_active":"2023-12-07T23:47:39.133797+0000","last_became_peered":"2023-12-07T23:47:39.133797+0000","last_unstale":"2023-12-08T00:18:07.511089+0000","last_undegraded":"2023-12-08T00:18:07.511089+0000","last_fullsized":"2023-12-08T00:18:07.511089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:38:14.932982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":996,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.861736+0000","last_change":"2023-12-07T23:47:20.151369+0000","last_active":"2023-12-08T00:18:06.861736+0000","last_peered":"2023-12-08T00:18:06.861736+0000","last_clean":"2023-12-08T00:18:06.861736+0000","last_became_active":"2023-12-07T23:47:20.132690+0000","last_became_peered":"2023-12-07T23:47:20.132690+0000","last_unstale":"2023-12-08T00:18:06.861736+0000","last_undegraded":"2023-12-08T00:18:06.861736+0000","last_fullsized":"2023-12-08T00:18:06.861736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":993,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.527725+0000","last_change":"2023-12-07T23:47:19.053664+0000","last_active":"2023-12-08T00:18:02.527725+0000","last_peered":"2023-12-08T00:18:02.527725+0000","last_clean":"2023-12-08T00:18:02.527725+0000","last_became_active":"2023-12-07T23:47:18.926939+0000","last_became_peered":"2023-12-07T23:47:18.926939+0000","last_unstale":"2023-12-08T00:18:02.527725+0000","last_undegraded":"2023-12-08T00:18:02.527725+0000","last_fullsized":"2023-12-08T00:18:02.527725+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:28:02.327836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1","version":"64'7","reported_seq":1009,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.982416+0000","last_change":"2023-12-07T23:47:15.440534+0000","last_active":"2023-12-08T00:18:07.982416+0000","last_peered":"2023-12-08T00:18:07.982416+0000","last_clean":"2023-12-08T00:18:07.982416+0000","last_became_active":"2023-12-07T23:47:15.386185+0000","last_became_peered":"2023-12-07T23:47:15.386185+0000","last_unstale":"2023-12-08T00:18:07.982416+0000","last_undegraded":"2023-12-08T00:18:07.982416+0000","last_fullsized":"2023-12-08T00:18:07.982416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":994,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.460347+0000","last_change":"2023-12-07T23:47:18.719911+0000","last_active":"2023-12-08T00:18:02.460347+0000","last_peered":"2023-12-08T00:18:02.460347+0000","last_clean":"2023-12-08T00:18:02.460347+0000","last_became_active":"2023-12-07T23:47:18.700591+0000","last_became_peered":"2023-12-07T23:47:18.700591+0000","last_unstale":"2023-12-08T00:18:02.460347+0000","last_undegraded":"2023-12-08T00:18:02.460347+0000","last_fullsized":"2023-12-08T00:18:02.460347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:15:06.752653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":993,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.641514+0000","last_change":"2023-12-07T23:47:16.106412+0000","last_active":"2023-12-08T00:18:02.641514+0000","last_peered":"2023-12-08T00:18:02.641514+0000","last_clean":"2023-12-08T00:18:02.641514+0000","last_became_active":"2023-12-07T23:47:16.101044+0000","last_became_peered":"2023-12-07T23:47:16.101044+0000","last_unstale":"2023-12-08T00:18:02.641514+0000","last_undegraded":"2023-12-08T00:18:02.641514+0000","last_fullsized":"2023-12-08T00:18:02.641514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:51:45.972358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":993,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.600736+0000","last_change":"2023-12-07T23:47:19.725523+0000","last_active":"2023-12-08T00:18:02.600736+0000","last_peered":"2023-12-08T00:18:02.600736+0000","last_clean":"2023-12-08T00:18:02.600736+0000","last_became_active":"2023-12-07T23:47:18.123992+0000","last_became_peered":"2023-12-07T23:47:18.123992+0000","last_unstale":"2023-12-08T00:18:02.600736+0000","last_undegraded":"2023-12-08T00:18:02.600736+0000","last_fullsized":"2023-12-08T00:18:02.600736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:05:44.580089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":996,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.875876+0000","last_change":"2023-12-07T23:47:19.649219+0000","last_active":"2023-12-08T00:18:06.875876+0000","last_peered":"2023-12-08T00:18:06.875876+0000","last_clean":"2023-12-08T00:18:06.875876+0000","last_became_active":"2023-12-07T23:47:19.507339+0000","last_became_peered":"2023-12-07T23:47:19.507339+0000","last_unstale":"2023-12-08T00:18:06.875876+0000","last_undegraded":"2023-12-08T00:18:06.875876+0000","last_fullsized":"2023-12-08T00:18:06.875876+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1011,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.180774+0000","last_change":"2023-12-07T23:48:06.632530+0000","last_active":"2023-12-08T00:18:02.180774+0000","last_peered":"2023-12-08T00:18:02.180774+0000","last_clean":"2023-12-08T00:18:02.180774+0000","last_became_active":"2023-12-07T23:48:06.627236+0000","last_became_peered":"2023-12-07T23:48:06.627236+0000","last_unstale":"2023-12-08T00:18:02.180774+0000","last_undegraded":"2023-12-08T00:18:02.180774+0000","last_fullsized":"2023-12-08T00:18:02.180774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:43:12.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1010,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.191720+0000","last_change":"2023-12-07T23:47:18.049591+0000","last_active":"2023-12-08T00:18:02.191720+0000","last_peered":"2023-12-08T00:18:02.191720+0000","last_clean":"2023-12-08T00:18:02.191720+0000","last_became_active":"2023-12-07T23:47:18.028936+0000","last_became_peered":"2023-12-07T23:47:18.028936+0000","last_unstale":"2023-12-08T00:18:02.191720+0000","last_undegraded":"2023-12-08T00:18:02.191720+0000","last_fullsized":"2023-12-08T00:18:02.191720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:11:43.305002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"1047'90","reported_seq":1132,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:10.087366+0000","last_change":"2023-12-08T00:18:10.088999+0000","last_active":"2023-12-08T00:18:10.087366+0000","last_peered":"2023-12-08T00:18:10.087366+0000","last_clean":"2023-12-08T00:18:10.087366+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-08T00:18:10.087366+0000","last_undegraded":"2023-12-08T00:18:10.087366+0000","last_fullsized":"2023-12-08T00:18:10.087366+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"1047'90","last_scrub_stamp":"2023-12-08T00:18:10.084655+0000","last_deep_scrub":"1047'90","last_deep_scrub_stamp":"2023-12-08T00:18:10.084655+0000","last_clean_scrub_stamp":"2023-12-08T00:18:10.084655+0000","objects_scrubbed":2,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:51:22.482725+0000","scrub_duration":0.204482201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1001,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.863281+0000","last_change":"2023-12-07T23:46:13.815701+0000","last_active":"2023-12-08T00:18:06.863281+0000","last_peered":"2023-12-08T00:18:06.863281+0000","last_clean":"2023-12-08T00:18:06.863281+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-08T00:18:06.863281+0000","last_undegraded":"2023-12-08T00:18:06.863281+0000","last_fullsized":"2023-12-08T00:18:06.863281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":995,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.995564+0000","last_change":"2023-12-07T23:47:18.113515+0000","last_active":"2023-12-08T00:18:07.995564+0000","last_peered":"2023-12-08T00:18:07.995564+0000","last_clean":"2023-12-08T00:18:07.995564+0000","last_became_active":"2023-12-07T23:47:18.104193+0000","last_became_peered":"2023-12-07T23:47:18.104193+0000","last_unstale":"2023-12-08T00:18:07.995564+0000","last_undegraded":"2023-12-08T00:18:07.995564+0000","last_fullsized":"2023-12-08T00:18:07.995564+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1001,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.850192+0000","last_change":"2023-12-07T23:46:13.418137+0000","last_active":"2023-12-08T00:18:06.850192+0000","last_peered":"2023-12-08T00:18:06.850192+0000","last_clean":"2023-12-08T00:18:06.850192+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-08T00:18:06.850192+0000","last_undegraded":"2023-12-08T00:18:06.850192+0000","last_fullsized":"2023-12-08T00:18:06.850192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1010,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:08.452893+0000","last_change":"2023-12-07T23:47:25.704244+0000","last_active":"2023-12-08T00:18:08.452893+0000","last_peered":"2023-12-08T00:18:08.452893+0000","last_clean":"2023-12-08T00:18:08.452893+0000","last_became_active":"2023-12-07T23:47:25.699009+0000","last_became_peered":"2023-12-07T23:47:25.699009+0000","last_unstale":"2023-12-08T00:18:08.452893+0000","last_undegraded":"2023-12-08T00:18:08.452893+0000","last_fullsized":"2023-12-08T00:18:08.452893+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1006,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.220802+0000","last_change":"2023-12-07T23:46:13.343801+0000","last_active":"2023-12-08T00:18:02.220802+0000","last_peered":"2023-12-08T00:18:02.220802+0000","last_clean":"2023-12-08T00:18:02.220802+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-08T00:18:02.220802+0000","last_undegraded":"2023-12-08T00:18:02.220802+0000","last_fullsized":"2023-12-08T00:18:02.220802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:42:20.201085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1009,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:08.454217+0000","last_change":"2023-12-07T23:47:26.181304+0000","last_active":"2023-12-08T00:18:08.454217+0000","last_peered":"2023-12-08T00:18:08.454217+0000","last_clean":"2023-12-08T00:18:08.454217+0000","last_became_active":"2023-12-07T23:47:26.173821+0000","last_became_peered":"2023-12-07T23:47:26.173821+0000","last_unstale":"2023-12-08T00:18:08.454217+0000","last_undegraded":"2023-12-08T00:18:08.454217+0000","last_fullsized":"2023-12-08T00:18:08.454217+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1001,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.866931+0000","last_change":"2023-12-07T23:46:13.873757+0000","last_active":"2023-12-08T00:18:06.866931+0000","last_peered":"2023-12-08T00:18:06.866931+0000","last_clean":"2023-12-08T00:18:06.866931+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-08T00:18:06.866931+0000","last_undegraded":"2023-12-08T00:18:06.866931+0000","last_fullsized":"2023-12-08T00:18:06.866931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1011,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:08.451592+0000","last_change":"2023-12-07T23:47:46.300836+0000","last_active":"2023-12-08T00:18:08.451592+0000","last_peered":"2023-12-08T00:18:08.451592+0000","last_clean":"2023-12-08T00:18:08.451592+0000","last_became_active":"2023-12-07T23:47:46.295177+0000","last_became_peered":"2023-12-07T23:47:46.295177+0000","last_unstale":"2023-12-08T00:18:08.451592+0000","last_undegraded":"2023-12-08T00:18:08.451592+0000","last_fullsized":"2023-12-08T00:18:08.451592+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":998,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.553256+0000","last_change":"2023-12-07T23:46:13.905749+0000","last_active":"2023-12-08T00:18:02.553256+0000","last_peered":"2023-12-08T00:18:02.553256+0000","last_clean":"2023-12-08T00:18:02.553256+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-08T00:18:02.553256+0000","last_undegraded":"2023-12-08T00:18:02.553256+0000","last_fullsized":"2023-12-08T00:18:02.553256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:45:13.859049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":992,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.497626+0000","last_change":"2023-12-07T23:47:18.243486+0000","last_active":"2023-12-08T00:18:07.497626+0000","last_peered":"2023-12-08T00:18:07.497626+0000","last_clean":"2023-12-08T00:18:07.497626+0000","last_became_active":"2023-12-07T23:47:18.206551+0000","last_became_peered":"2023-12-07T23:47:18.206551+0000","last_unstale":"2023-12-08T00:18:07.497626+0000","last_undegraded":"2023-12-08T00:18:07.497626+0000","last_fullsized":"2023-12-08T00:18:07.497626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:33:27.429730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1001,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.874333+0000","last_change":"2023-12-07T23:46:13.447293+0000","last_active":"2023-12-08T00:18:06.874333+0000","last_peered":"2023-12-08T00:18:06.874333+0000","last_clean":"2023-12-08T00:18:06.874333+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-08T00:18:06.874333+0000","last_undegraded":"2023-12-08T00:18:06.874333+0000","last_fullsized":"2023-12-08T00:18:06.874333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":992,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.518954+0000","last_change":"2023-12-07T23:47:15.786542+0000","last_active":"2023-12-08T00:18:02.518954+0000","last_peered":"2023-12-08T00:18:02.518954+0000","last_clean":"2023-12-08T00:18:02.518954+0000","last_became_active":"2023-12-07T23:47:15.770063+0000","last_became_peered":"2023-12-07T23:47:15.770063+0000","last_unstale":"2023-12-08T00:18:02.518954+0000","last_undegraded":"2023-12-08T00:18:02.518954+0000","last_fullsized":"2023-12-08T00:18:02.518954+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:18:15.429980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1015,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.193315+0000","last_change":"2023-12-07T23:46:13.656681+0000","last_active":"2023-12-08T00:18:02.193315+0000","last_peered":"2023-12-08T00:18:02.193315+0000","last_clean":"2023-12-08T00:18:02.193315+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-08T00:18:02.193315+0000","last_undegraded":"2023-12-08T00:18:02.193315+0000","last_fullsized":"2023-12-08T00:18:02.193315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:29.359895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":993,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.526105+0000","last_change":"2023-12-07T23:47:15.701780+0000","last_active":"2023-12-08T00:18:02.526105+0000","last_peered":"2023-12-08T00:18:02.526105+0000","last_clean":"2023-12-08T00:18:02.526105+0000","last_became_active":"2023-12-07T23:47:15.693158+0000","last_became_peered":"2023-12-07T23:47:15.693158+0000","last_unstale":"2023-12-08T00:18:02.526105+0000","last_undegraded":"2023-12-08T00:18:02.526105+0000","last_fullsized":"2023-12-08T00:18:02.526105+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:04:12.790526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1000,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.877555+0000","last_change":"2023-12-07T23:46:13.620414+0000","last_active":"2023-12-08T00:18:06.877555+0000","last_peered":"2023-12-08T00:18:06.877555+0000","last_clean":"2023-12-08T00:18:06.877555+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-08T00:18:06.877555+0000","last_undegraded":"2023-12-08T00:18:06.877555+0000","last_fullsized":"2023-12-08T00:18:06.877555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:46:11.191893+0000","last_clean_scrub_stamp":"2023-12-07T23:46:11.191893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1000,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.191057+0000","last_change":"2023-12-07T23:47:21.241228+0000","last_active":"2023-12-08T00:18:02.191057+0000","last_peered":"2023-12-08T00:18:02.191057+0000","last_clean":"2023-12-08T00:18:02.191057+0000","last_became_active":"2023-12-07T23:47:21.229799+0000","last_became_peered":"2023-12-07T23:47:21.229799+0000","last_unstale":"2023-12-08T00:18:02.191057+0000","last_undegraded":"2023-12-08T00:18:02.191057+0000","last_fullsized":"2023-12-08T00:18:02.191057+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:30:12.922388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":994,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.985445+0000","last_change":"2023-12-07T23:47:36.622692+0000","last_active":"2023-12-08T00:18:07.985445+0000","last_peered":"2023-12-08T00:18:07.985445+0000","last_clean":"2023-12-08T00:18:07.985445+0000","last_became_active":"2023-12-07T23:47:36.616695+0000","last_became_peered":"2023-12-07T23:47:36.616695+0000","last_unstale":"2023-12-08T00:18:07.985445+0000","last_undegraded":"2023-12-08T00:18:07.985445+0000","last_fullsized":"2023-12-08T00:18:07.985445+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":994,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:08.005272+0000","last_change":"2023-12-07T23:47:36.574414+0000","last_active":"2023-12-08T00:18:08.005272+0000","last_peered":"2023-12-08T00:18:08.005272+0000","last_clean":"2023-12-08T00:18:08.005272+0000","last_became_active":"2023-12-07T23:47:36.567130+0000","last_became_peered":"2023-12-07T23:47:36.567130+0000","last_unstale":"2023-12-08T00:18:08.005272+0000","last_undegraded":"2023-12-08T00:18:08.005272+0000","last_fullsized":"2023-12-08T00:18:08.005272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":994,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.860190+0000","last_change":"2023-12-07T23:47:37.286320+0000","last_active":"2023-12-08T00:18:06.860190+0000","last_peered":"2023-12-08T00:18:06.860190+0000","last_clean":"2023-12-08T00:18:06.860190+0000","last_became_active":"2023-12-07T23:47:37.279080+0000","last_became_peered":"2023-12-07T23:47:37.279080+0000","last_unstale":"2023-12-08T00:18:06.860190+0000","last_undegraded":"2023-12-08T00:18:06.860190+0000","last_fullsized":"2023-12-08T00:18:06.860190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":995,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.880071+0000","last_change":"2023-12-07T23:47:21.394853+0000","last_active":"2023-12-08T00:18:06.880071+0000","last_peered":"2023-12-08T00:18:06.880071+0000","last_clean":"2023-12-08T00:18:06.880071+0000","last_became_active":"2023-12-07T23:47:21.389382+0000","last_became_peered":"2023-12-07T23:47:21.389382+0000","last_unstale":"2023-12-08T00:18:06.880071+0000","last_undegraded":"2023-12-08T00:18:06.880071+0000","last_fullsized":"2023-12-08T00:18:06.880071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1009,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:08.458537+0000","last_change":"2023-12-07T23:47:20.975428+0000","last_active":"2023-12-08T00:18:08.458537+0000","last_peered":"2023-12-08T00:18:08.458537+0000","last_clean":"2023-12-08T00:18:08.458537+0000","last_became_active":"2023-12-07T23:47:20.968485+0000","last_became_peered":"2023-12-07T23:47:20.968485+0000","last_unstale":"2023-12-08T00:18:08.458537+0000","last_undegraded":"2023-12-08T00:18:08.458537+0000","last_fullsized":"2023-12-08T00:18:08.458537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":990,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.504466+0000","last_change":"2023-12-07T23:47:21.404597+0000","last_active":"2023-12-08T00:18:07.504466+0000","last_peered":"2023-12-08T00:18:07.504466+0000","last_clean":"2023-12-08T00:18:07.504466+0000","last_became_active":"2023-12-07T23:47:21.395304+0000","last_became_peered":"2023-12-07T23:47:21.395304+0000","last_unstale":"2023-12-08T00:18:07.504466+0000","last_undegraded":"2023-12-08T00:18:07.504466+0000","last_fullsized":"2023-12-08T00:18:07.504466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:02:02.413893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1008,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:08.459962+0000","last_change":"2023-12-07T23:47:38.795823+0000","last_active":"2023-12-08T00:18:08.459962+0000","last_peered":"2023-12-08T00:18:08.459962+0000","last_clean":"2023-12-08T00:18:08.459962+0000","last_became_active":"2023-12-07T23:47:38.790323+0000","last_became_peered":"2023-12-07T23:47:38.790323+0000","last_unstale":"2023-12-08T00:18:08.459962+0000","last_undegraded":"2023-12-08T00:18:08.459962+0000","last_fullsized":"2023-12-08T00:18:08.459962+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":995,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:06.852466+0000","last_change":"2023-12-07T23:47:37.187628+0000","last_active":"2023-12-08T00:18:06.852466+0000","last_peered":"2023-12-08T00:18:06.852466+0000","last_clean":"2023-12-08T00:18:06.852466+0000","last_became_active":"2023-12-07T23:47:37.180489+0000","last_became_peered":"2023-12-07T23:47:37.180489+0000","last_unstale":"2023-12-08T00:18:06.852466+0000","last_undegraded":"2023-12-08T00:18:06.852466+0000","last_fullsized":"2023-12-08T00:18:06.852466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1010,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.163354+0000","last_change":"2023-12-07T23:48:09.211978+0000","last_active":"2023-12-08T00:18:02.163354+0000","last_peered":"2023-12-08T00:18:02.163354+0000","last_clean":"2023-12-08T00:18:02.163354+0000","last_became_active":"2023-12-07T23:48:09.186728+0000","last_became_peered":"2023-12-07T23:48:09.186728+0000","last_unstale":"2023-12-08T00:18:02.163354+0000","last_undegraded":"2023-12-08T00:18:02.163354+0000","last_fullsized":"2023-12-08T00:18:02.163354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:09:04.978015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":991,"reported_epoch":1047,"state":"active+clean","last_fresh":"2023-12-08T00:18:07.508362+0000","last_change":"2023-12-07T23:47:20.131095+0000","last_active":"2023-12-08T00:18:07.508362+0000","last_peered":"2023-12-08T00:18:07.508362+0000","last_clean":"2023-12-08T00:18:07.508362+0000","last_became_active":"2023-12-07T23:47:20.123312+0000","last_became_peered":"2023-12-07T23:47:20.123312+0000","last_unstale":"2023-12-08T00:18:07.508362+0000","last_undegraded":"2023-12-08T00:18:07.508362+0000","last_fullsized":"2023-12-08T00:18:07.508362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:27:53.246959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1010,"reported_epoch":1046,"state":"active+clean","last_fresh":"2023-12-08T00:18:02.196019+0000","last_change":"2023-12-07T23:47:18.960597+0000","last_active":"2023-12-08T00:18:02.196019+0000","last_peered":"2023-12-08T00:18:02.196019+0000","last_clean":"2023-12-08T00:18:02.196019+0000","last_became_active":"2023-12-07T23:47:18.946564+0000","last_became_peered":"2023-12-07T23:47:18.946564+0000","last_unstale":"2023-12-08T00:18:02.196019+0000","last_undegraded":"2023-12-08T00:18:02.196019+0000","last_fullsized":"2023-12-08T00:18:02.196019+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:55:18.226923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":2,"acting":2,"num_store_stats":2},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477121,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81916,"kb_used_data":6088,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289924,"statfs":{"total":96636764160,"available":96552882176,"internally_reserved":0,"allocated":6234112,"data_stored":2932452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":8.3040000000000003,"5min":8.2520000000000007,"15min":15.449999999999999},"min":{"1min":4.4939999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"max":{"1min":20.571999999999999,"5min":20.571999999999999,"15min":272.77100000000002},"last":9.0719999999999992},{"interface":"front","average":{"1min":11.718,"5min":10.836,"15min":18.363},"min":{"1min":5.5990000000000002,"5min":4.9900000000000002,"15min":4.9649999999999999},"max":{"1min":47.829999999999998,"5min":47.829999999999998,"15min":288.42000000000002},"last":11.749000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.33,"5min":11.922000000000001,"15min":21.003},"min":{"1min":6.2279999999999998,"5min":6.2279999999999998,"15min":5.7649999999999997},"max":{"1min":73.218999999999994,"5min":73.218999999999994,"15min":320.57900000000001},"last":13.147},{"interface":"front","average":{"1min":16.672000000000001,"5min":13.327,"15min":23.423999999999999},"min":{"1min":9.1400000000000006,"5min":6.7279999999999998,"15min":5.5670000000000002},"max":{"1min":91.882000000000005,"5min":91.882000000000005,"15min":461.51799999999997},"last":38.854999999999997}]},{"osd":2,"last update":"Fri Dec 8 00:17:38 2023","interfaces":[{"interface":"back","average":{"1min":11.853,"5min":9.7810000000000006,"15min":15.744},"min":{"1min":7.343,"5min":6.0389999999999997,"15min":5.6959999999999997},"max":{"1min":66.941999999999993,"5min":66.941999999999993,"15min":246.47499999999999},"last":28.959},{"interface":"front","average":{"1min":17.085000000000001,"5min":14.343,"15min":22.997},"min":{"1min":8.7669999999999995,"5min":8.7279999999999998,"15min":7.3079999999999998},"max":{"1min":69.078999999999994,"5min":69.078999999999994,"15min":356.02100000000002},"last":37.097000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":17.209,"5min":14.24,"15min":19.495999999999999},"min":{"1min":7.7300000000000004,"5min":7.048,"15min":6.7690000000000001},"max":{"1min":107.04900000000001,"5min":107.04900000000001,"15min":180.76400000000001},"last":33.478999999999999},{"interface":"front","average":{"1min":16.361000000000001,"5min":14.583,"15min":22.693000000000001},"min":{"1min":10.125,"5min":6.5629999999999997,"15min":6.2990000000000004},"max":{"1min":62.020000000000003,"5min":62.020000000000003,"15min":223.91300000000001},"last":34.676000000000002}]},{"osd":5,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":14.823,"5min":15.294,"15min":24.542000000000002},"min":{"1min":10.445,"5min":8.9920000000000009,"15min":7.2889999999999997},"max":{"1min":19.890000000000001,"5min":29.309999999999999,"15min":251.892},"last":13.855},{"interface":"front","average":{"1min":29.396999999999998,"5min":19.707000000000001,"15min":28.649999999999999},"min":{"1min":13.223000000000001,"5min":11.711,"15min":10.568},"max":{"1min":265.69799999999998,"5min":265.69799999999998,"15min":279.791},"last":39.506999999999998}]},{"osd":6,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":21.244,"5min":14.326000000000001,"15min":19.547000000000001},"min":{"1min":7.2850000000000001,"5min":7.2850000000000001,"15min":7.2850000000000001},"max":{"1min":214.774,"5min":214.774,"15min":214.774},"last":10.486000000000001},{"interface":"front","average":{"1min":26.649999999999999,"5min":17.835000000000001,"15min":24.134},"min":{"1min":12.226000000000001,"5min":10.428000000000001,"15min":8.1440000000000001},"max":{"1min":229.54300000000001,"5min":229.54300000000001,"15min":229.54300000000001},"last":32.823}]},{"osd":7,"last update":"Fri Dec 8 00:17:16 2023","interfaces":[{"interface":"back","average":{"1min":16.917000000000002,"5min":17.433,"15min":28.428000000000001},"min":{"1min":8.923,"5min":8.923,"15min":8.923},"max":{"1min":21.625,"5min":27.763999999999999,"15min":281.31},"last":40.479999999999997},{"interface":"front","average":{"1min":14.558999999999999,"5min":14.391999999999999,"15min":23.718},"min":{"1min":9.9049999999999994,"5min":8.5600000000000005,"15min":7.649},"max":{"1min":23.234999999999999,"5min":23.234999999999999,"15min":255.72499999999999},"last":35.935000000000002}]}]},{"osd":2,"up_from":16,"seq":68719477122,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69204,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302636,"statfs":{"total":96636764160,"available":96565899264,"internally_reserved":0,"allocated":5996544,"data_stored":2800717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5220000000000002,"5min":8.8870000000000005,"15min":15.034000000000001},"min":{"1min":4.8140000000000001,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":20.408000000000001,"5min":20.408000000000001,"15min":249.42699999999999},"last":7.1609999999999996},{"interface":"front","average":{"1min":9.9410000000000007,"5min":9.4969999999999999,"15min":15.816000000000001},"min":{"1min":5.9480000000000004,"5min":5.8230000000000004,"15min":5.1310000000000002},"max":{"1min":19.068999999999999,"5min":19.068999999999999,"15min":241.45599999999999},"last":30.783000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":12.634,"5min":11.613,"15min":19.722000000000001},"min":{"1min":8.1159999999999997,"5min":6.8179999999999996,"15min":6.766},"max":{"1min":19.122,"5min":19.122,"15min":352.21100000000001},"last":32.902000000000001},{"interface":"front","average":{"1min":11.063000000000001,"5min":10.92,"15min":17.779},"min":{"1min":8.3200000000000003,"5min":6.2720000000000002,"15min":5.9109999999999996},"max":{"1min":14.936,"5min":18.652000000000001,"15min":327.81099999999998},"last":8.4710000000000001}]},{"osd":3,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.925000000000001,"5min":10.75,"15min":19.754000000000001},"min":{"1min":7.798,"5min":6.2770000000000001,"15min":6.1550000000000002},"max":{"1min":15.944000000000001,"5min":19.123999999999999,"15min":287.56299999999999},"last":25.300000000000001},{"interface":"front","average":{"1min":14.613,"5min":13.928000000000001,"15min":21.443999999999999},"min":{"1min":8.9900000000000002,"5min":8.3059999999999992,"15min":7.149},"max":{"1min":17.920000000000002,"5min":20.739000000000001,"15min":255.334},"last":24.643000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":15.124000000000001,"5min":15.16,"15min":23.259},"min":{"1min":8.9789999999999992,"5min":8.9220000000000006,"15min":8.1539999999999999},"max":{"1min":19.739999999999998,"5min":20.571000000000002,"15min":301.12799999999999},"last":34.311},{"interface":"front","average":{"1min":12.436999999999999,"5min":12.302,"15min":20.571000000000002},"min":{"1min":6.7450000000000001,"5min":6.7450000000000001,"15min":6.2649999999999997},"max":{"1min":19.678999999999998,"5min":23.859999999999999,"15min":348.38099999999997},"last":10.023}]},{"osd":5,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":16.494,"15min":24.294},"min":{"1min":14.236000000000001,"5min":9.0410000000000004,"15min":8.5470000000000006},"max":{"1min":20.163,"5min":22.094000000000001,"15min":289.18700000000001},"last":31.372},{"interface":"front","average":{"1min":14.500999999999999,"5min":14.137,"15min":20.013000000000002},"min":{"1min":9.3870000000000005,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":20.331,"5min":20.667000000000002,"15min":248.63499999999999},"last":21.904}]},{"osd":6,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":16.123000000000001,"5min":15.634,"15min":21.067},"min":{"1min":10.611000000000001,"5min":9.9610000000000003,"15min":7.4240000000000004},"max":{"1min":19.946000000000002,"5min":21.585000000000001,"15min":349.41399999999999},"last":22.523},{"interface":"front","average":{"1min":11.491,"5min":12.48,"15min":20.643999999999998},"min":{"1min":7.431,"5min":7.423,"15min":7.423},"max":{"1min":19.212,"5min":21.007000000000001,"15min":231.422},"last":35.481000000000002}]},{"osd":7,"last update":"Fri Dec 8 00:17:12 2023","interfaces":[{"interface":"back","average":{"1min":13.156000000000001,"5min":14.472,"15min":24.436},"min":{"1min":8.6010000000000009,"5min":8.6010000000000009,"15min":8.6010000000000009},"max":{"1min":20.013999999999999,"5min":20.428000000000001,"15min":350.52199999999999},"last":29.949000000000002},{"interface":"front","average":{"1min":16.969000000000001,"5min":17.178000000000001,"15min":24.536000000000001},"min":{"1min":11.411,"5min":10.608000000000001,"15min":10.475},"max":{"1min":20.597000000000001,"5min":22.510999999999999,"15min":235.19200000000001},"last":23.507999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477121,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69436,"kb_used_data":6088,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302404,"statfs":{"total":96636764160,"available":96565661696,"internally_reserved":0,"allocated":6234112,"data_stored":2932452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 8 00:17:29 2023","interfaces":[{"interface":"back","average":{"1min":10.356,"5min":9.7119999999999997,"15min":18.596},"min":{"1min":5.2789999999999999,"5min":4.827,"15min":4.5830000000000002},"max":{"1min":38.834000000000003,"5min":38.834000000000003,"15min":317.99599999999998},"last":5.4109999999999996},{"interface":"front","average":{"1min":11.698,"5min":9.2210000000000001,"15min":17.318999999999999},"min":{"1min":6.6840000000000002,"5min":4.9139999999999997,"15min":4.9139999999999997},"max":{"1min":49.262,"5min":49.262,"15min":313.14400000000001},"last":8.9719999999999995}]},{"osd":2,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":30.468,"5min":16.053000000000001,"15min":20.640000000000001},"min":{"1min":8.9280000000000008,"5min":7.1600000000000001,"15min":6.4740000000000002},"max":{"1min":196.96700000000001,"5min":196.96700000000001,"15min":228.02500000000001},"last":13.983000000000001},{"interface":"front","average":{"1min":23.956,"5min":14.147,"15min":18.375},"min":{"1min":6.8819999999999997,"5min":5.7530000000000001,"15min":5.6059999999999999},"max":{"1min":168.91300000000001,"5min":168.91300000000001,"15min":197.78899999999999},"last":13.065}]},{"osd":3,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":14.613,"5min":13.257,"15min":22.777999999999999},"min":{"1min":10.614000000000001,"5min":8.0500000000000007,"15min":7.0069999999999997},"max":{"1min":19.207999999999998,"5min":19.207999999999998,"15min":239.50999999999999},"last":9.3490000000000002},{"interface":"front","average":{"1min":16.495999999999999,"5min":16.079000000000001,"15min":24.495999999999999},"min":{"1min":10.835000000000001,"5min":10.667,"15min":9.141},"max":{"1min":20.774000000000001,"5min":29.132999999999999,"15min":241.839},"last":17.594000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:08 2023","interfaces":[{"interface":"back","average":{"1min":8.8249999999999993,"5min":8.9369999999999994,"15min":14.958},"min":{"1min":6.4029999999999996,"5min":5.5570000000000004,"15min":5.5430000000000001},"max":{"1min":22.925000000000001,"5min":22.925000000000001,"15min":211.452},"last":9.9559999999999995},{"interface":"front","average":{"1min":16.763999999999999,"5min":16.686,"15min":22.658000000000001},"min":{"1min":10.212999999999999,"5min":10.212999999999999,"15min":9.3030000000000008},"max":{"1min":24.173999999999999,"5min":29.969000000000001,"15min":212.459},"last":19.443999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:22 2023","interfaces":[{"interface":"back","average":{"1min":21.459,"5min":14.852,"15min":19.033999999999999},"min":{"1min":8.6750000000000007,"5min":7.8970000000000002,"15min":5.9740000000000002},"max":{"1min":175.68100000000001,"5min":175.68100000000001,"15min":175.68100000000001},"last":15.385999999999999},{"interface":"front","average":{"1min":22.960000000000001,"5min":13.959,"15min":19.902000000000001},"min":{"1min":8.1950000000000003,"5min":6.7789999999999999,"15min":6.6660000000000004},"max":{"1min":186.625,"5min":186.625,"15min":186.625},"last":10.718}]},{"osd":6,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":17.789999999999999,"15min":26.085000000000001},"min":{"1min":11.406000000000001,"5min":11.406000000000001,"15min":10.641999999999999},"max":{"1min":26.757999999999999,"5min":30.727,"15min":227.892},"last":18.657},{"interface":"front","average":{"1min":16.129999999999999,"5min":14.592000000000001,"15min":20.123999999999999},"min":{"1min":10.122,"5min":8.2919999999999998,"15min":8.2919999999999998},"max":{"1min":44.432000000000002,"5min":44.432000000000002,"15min":153.59700000000001},"last":14.752000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:17:21 2023","interfaces":[{"interface":"back","average":{"1min":17.475000000000001,"5min":16.189,"15min":24.940000000000001},"min":{"1min":11.638,"5min":9.3550000000000004,"15min":8.9369999999999994},"max":{"1min":39.835999999999999,"5min":39.835999999999999,"15min":264.44600000000003},"last":16.599},{"interface":"front","average":{"1min":14.044,"5min":12.993,"15min":20.210000000000001},"min":{"1min":7.8209999999999997,"5min":7.5350000000000001,"15min":7.5},"max":{"1min":41.091000000000001,"5min":41.091000000000001,"15min":243.84899999999999},"last":12.164999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542527,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69600,"kb_used_data":6252,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302240,"statfs":{"total":96636764160,"available":96565493760,"internally_reserved":0,"allocated":6402048,"data_stored":3198562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.6829999999999998,"5min":9.3740000000000006,"15min":17.443000000000001},"min":{"1min":5.8540000000000001,"5min":5.6449999999999996,"15min":4.2539999999999996},"max":{"1min":17.231000000000002,"5min":18.986999999999998,"15min":324.46600000000001},"last":7.9960000000000004},{"interface":"front","average":{"1min":14.244,"5min":13.406000000000001,"15min":22.675000000000001},"min":{"1min":7.2469999999999999,"5min":6.4950000000000001,"15min":5.5369999999999999},"max":{"1min":21.832000000000001,"5min":21.832000000000001,"15min":342.39299999999997},"last":11.114000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":9.9380000000000006,"5min":9.7550000000000008,"15min":20.015000000000001},"min":{"1min":5.6289999999999996,"5min":4.7409999999999997,"15min":4.7409999999999997},"max":{"1min":15.329000000000001,"5min":18.332000000000001,"15min":309.75999999999999},"last":7.0999999999999996},{"interface":"front","average":{"1min":13.932,"5min":13.237,"15min":23.968},"min":{"1min":7.7919999999999998,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":19.77,"5min":22.640999999999998,"15min":343.49900000000002},"last":17.283000000000001}]},{"osd":3,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":13.263,"5min":14.33,"15min":22.765000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"max":{"1min":17.984999999999999,"5min":20.395,"15min":345.07600000000002},"last":18.073},{"interface":"front","average":{"1min":10.297000000000001,"5min":9.077,"15min":15.353},"min":{"1min":7.5209999999999999,"5min":6.1050000000000004,"15min":5.3090000000000002},"max":{"1min":15.089,"5min":16.981999999999999,"15min":174.20599999999999},"last":10.441000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.705,"5min":11.641999999999999,"15min":19.434999999999999},"min":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.0419999999999998},"max":{"1min":18.649999999999999,"5min":19.026,"15min":290.358},"last":11.863},{"interface":"front","average":{"1min":12.657999999999999,"5min":12.154999999999999,"15min":19.594999999999999},"min":{"1min":7.8029999999999999,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":17.945,"5min":19.114000000000001,"15min":203.929},"last":13.212999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":32.521999999999998,"5min":19.515000000000001,"15min":24.640999999999998},"min":{"1min":10.952999999999999,"5min":10.938000000000001,"15min":7.7690000000000001},"max":{"1min":333.11799999999999,"5min":333.11799999999999,"15min":371.50599999999997},"last":19.318999999999999},{"interface":"front","average":{"1min":28.724,"5min":15.845000000000001,"15min":19.704000000000001},"min":{"1min":9.0389999999999997,"5min":7.6150000000000002,"15min":7.282},"max":{"1min":298.12099999999998,"5min":298.12099999999998,"15min":298.12099999999998},"last":20.178000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":14.052,"5min":13.733000000000001,"15min":19.829000000000001},"min":{"1min":9.0879999999999992,"5min":7.3490000000000002,"15min":7.3490000000000002},"max":{"1min":19.085000000000001,"5min":24.93,"15min":292.72300000000001},"last":13.675000000000001},{"interface":"front","average":{"1min":15.968999999999999,"5min":14.984,"15min":22.007999999999999},"min":{"1min":10.695,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":18.954999999999998,"5min":26.942,"15min":173.17599999999999},"last":14.41}]},{"osd":7,"last update":"Fri Dec 8 00:18:04 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":16.228999999999999,"15min":26.417000000000002},"min":{"1min":10.32,"5min":9.3789999999999996,"15min":9.3789999999999996},"max":{"1min":94.143000000000001,"5min":94.143000000000001,"15min":350.50999999999999},"last":15.645},{"interface":"front","average":{"1min":24.823,"5min":18.376999999999999,"15min":24.126000000000001},"min":{"1min":12.754,"5min":12.005000000000001,"15min":8.3659999999999997},"max":{"1min":90.555999999999997,"5min":90.555999999999997,"15min":304.64499999999998},"last":16.321000000000002}]}]},{"osd":7,"up_from":13,"seq":55834575234,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69204,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302636,"statfs":{"total":96636764160,"available":96565899264,"internally_reserved":0,"allocated":5996544,"data_stored":2800717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.7469999999999999,"5min":9.6129999999999995,"15min":18.939},"min":{"1min":5.4649999999999999,"5min":5.2800000000000002,"15min":4.2489999999999997},"max":{"1min":13.460000000000001,"5min":17.106999999999999,"15min":259.01499999999999},"last":12.249000000000001},{"interface":"front","average":{"1min":14.004,"5min":14.007999999999999,"15min":23.878},"min":{"1min":6.1369999999999996,"5min":6.1369999999999996,"15min":6.016},"max":{"1min":22.023,"5min":22.023,"15min":292.73899999999998},"last":15.662000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":9.0190000000000001,"5min":9.6690000000000005,"15min":19.279},"min":{"1min":4.5,"5min":4.5,"15min":4.5},"max":{"1min":17.582999999999998,"5min":20.797000000000001,"15min":414.57100000000003},"last":6.2939999999999996},{"interface":"front","average":{"1min":10.18,"5min":10.154999999999999,"15min":17.631},"min":{"1min":6.9130000000000003,"5min":6.4880000000000004,"15min":5.7709999999999999},"max":{"1min":16.693999999999999,"5min":19.157,"15min":333.53800000000001},"last":11.178000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":11.805,"5min":12.065,"15min":21.433},"min":{"1min":7.7130000000000001,"5min":6.75,"15min":6.75},"max":{"1min":18.497,"5min":19.593,"15min":286.63499999999999},"last":7.3449999999999998},{"interface":"front","average":{"1min":14.807,"5min":12.464,"15min":21.170000000000002},"min":{"1min":9.2910000000000004,"5min":7.0540000000000003,"15min":6.1280000000000001},"max":{"1min":19.707999999999998,"5min":21.053000000000001,"15min":299.54300000000001},"last":13.613}]},{"osd":3,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.760999999999999,"15min":24.027000000000001},"min":{"1min":8.9309999999999992,"5min":7.0810000000000004,"15min":6.6289999999999996},"max":{"1min":16.792000000000002,"5min":22.248000000000001,"15min":391.23200000000003},"last":12.962999999999999},{"interface":"front","average":{"1min":16.058,"5min":16.571000000000002,"15min":26.021999999999998},"min":{"1min":10.843,"5min":10.125,"15min":8.3949999999999996},"max":{"1min":20.483000000000001,"5min":22.477,"15min":362.20299999999997},"last":17.210000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.838999999999999,"5min":17.010999999999999,"15min":23.010000000000002},"min":{"1min":12.465999999999999,"5min":7.6710000000000003,"15min":7.6710000000000003},"max":{"1min":20.530000000000001,"5min":22.053000000000001,"15min":293.358},"last":14.263999999999999},{"interface":"front","average":{"1min":14.430999999999999,"5min":13.443,"15min":20.056999999999999},"min":{"1min":9.4139999999999997,"5min":7.7149999999999999,"15min":6.6769999999999996},"max":{"1min":19.288,"5min":19.962,"15min":275.31799999999998},"last":10.547000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":17.608000000000001,"5min":17.893000000000001,"15min":26.475000000000001},"min":{"1min":11.775,"5min":9.6720000000000006,"15min":7.7380000000000004},"max":{"1min":21.373000000000001,"5min":23.498000000000001,"15min":294.56400000000002},"last":18.117000000000001},{"interface":"front","average":{"1min":9.9760000000000009,"5min":11.784000000000001,"15min":22.891999999999999},"min":{"1min":6.3739999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"max":{"1min":17.087,"5min":21.634,"15min":280.83600000000001},"last":8.4800000000000004}]},{"osd":6,"last update":"Fri Dec 8 00:17:09 2023","interfaces":[{"interface":"back","average":{"1min":16.66,"5min":15.532999999999999,"15min":24.460000000000001},"min":{"1min":11.808999999999999,"5min":10.372,"15min":8.5579999999999998},"max":{"1min":21.218,"5min":22.132000000000001,"15min":289.798},"last":14.923999999999999},{"interface":"front","average":{"1min":12.567,"5min":13.994999999999999,"15min":23.161999999999999},"min":{"1min":8.3300000000000001,"5min":7.5129999999999999,"15min":7.5129999999999999},"max":{"1min":18.094000000000001,"5min":20.853999999999999,"15min":303.30900000000003},"last":9.0920000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575233,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86080,"kb_used_data":6092,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94285760,"statfs":{"total":96636764160,"available":96548618240,"internally_reserved":0,"allocated":6238208,"data_stored":2932457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:18 2023","interfaces":[{"interface":"back","average":{"1min":10.628,"5min":10.327,"15min":14.347},"min":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":4.1920000000000002},"max":{"1min":15.675000000000001,"5min":18.510000000000002,"15min":168.34100000000001},"last":23.716999999999999},{"interface":"front","average":{"1min":9.1110000000000007,"5min":8.9320000000000004,"15min":13.920999999999999},"min":{"1min":5.6600000000000001,"5min":5.2050000000000001,"15min":5.2050000000000001},"max":{"1min":15.646000000000001,"5min":19.010999999999999,"15min":172.452},"last":10.488}]},{"osd":1,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":12.702999999999999,"5min":13.539,"15min":16.152000000000001},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.1870000000000003},"max":{"1min":18.241,"5min":20.835000000000001,"15min":206.43899999999999},"last":11.587999999999999},{"interface":"front","average":{"1min":16.012,"5min":16.016999999999999,"15min":19.134},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":7.2080000000000002},"max":{"1min":35.750999999999998,"5min":35.750999999999998,"15min":155.94},"last":21.329999999999998}]},{"osd":2,"last update":"Fri Dec 8 00:18:07 2023","interfaces":[{"interface":"back","average":{"1min":10.789,"5min":11.645,"15min":15.612},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":5.8940000000000001},"max":{"1min":21.337,"5min":21.337,"15min":168.23599999999999},"last":9.5},{"interface":"front","average":{"1min":9.5329999999999995,"5min":9.9109999999999996,"15min":15.417999999999999},"min":{"1min":6.3979999999999997,"5min":4.8540000000000001,"15min":4.8540000000000001},"max":{"1min":20.876999999999999,"5min":20.876999999999999,"15min":352.52800000000002},"last":6.9139999999999997}]},{"osd":3,"last update":"Fri Dec 8 00:17:39 2023","interfaces":[{"interface":"back","average":{"1min":11.417999999999999,"5min":11.669,"15min":17.757999999999999},"min":{"1min":6.867,"5min":6.867,"15min":5.8789999999999996},"max":{"1min":21.459,"5min":21.459,"15min":286.92399999999998},"last":30.916},{"interface":"front","average":{"1min":17.001000000000001,"5min":15.720000000000001,"15min":22.109000000000002},"min":{"1min":9.8889999999999993,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":36.771000000000001,"5min":36.771000000000001,"15min":265.44400000000002},"last":34.753999999999998}]},{"osd":4,"last update":"Fri Dec 8 00:17:46 2023","interfaces":[{"interface":"back","average":{"1min":13.489000000000001,"5min":13.015000000000001,"15min":18.609000000000002},"min":{"1min":7.3319999999999999,"5min":6.7389999999999999,"15min":6.7389999999999999},"max":{"1min":33.700000000000003,"5min":33.700000000000003,"15min":353.452},"last":8.1319999999999997},{"interface":"front","average":{"1min":17.852,"5min":16.946000000000002,"15min":23.024000000000001},"min":{"1min":9.7469999999999999,"5min":9.7469999999999999,"15min":8.8499999999999996},"max":{"1min":37.723999999999997,"5min":37.723999999999997,"15min":341.87299999999999},"last":33.415999999999997}]},{"osd":5,"last update":"Fri Dec 8 00:18:07 2023","interfaces":[{"interface":"back","average":{"1min":20.192,"5min":18.701000000000001,"15min":23.984000000000002},"min":{"1min":8.9990000000000006,"5min":8.9990000000000006,"15min":8.9990000000000006},"max":{"1min":70.762,"5min":70.762,"15min":158.316},"last":35.978000000000002},{"interface":"front","average":{"1min":15.691000000000001,"5min":12.763,"15min":17.699999999999999},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.056},"max":{"1min":66.159000000000006,"5min":66.159000000000006,"15min":157.48500000000001},"last":37.156999999999996}]},{"osd":7,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":13.949999999999999,"5min":14.15,"15min":18.186},"min":{"1min":8.5719999999999992,"5min":8.5719999999999992,"15min":8.4930000000000003},"max":{"1min":30.861000000000001,"5min":30.861000000000001,"15min":152.49600000000001},"last":37.810000000000002},{"interface":"front","average":{"1min":18.798999999999999,"5min":18.960000000000001,"15min":23.832999999999998},"min":{"1min":11.927,"5min":11.927,"15min":11.388},"max":{"1min":38.762,"5min":38.762,"15min":299.93099999999998},"last":35.345999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575232,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74208,"kb_used_data":6700,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297632,"statfs":{"total":96636764160,"available":96560775168,"internally_reserved":0,"allocated":6860800,"data_stored":3458318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":8.5069999999999997,"15min":14.414999999999999},"min":{"1min":4.4740000000000002,"5min":4.226,"15min":4.226},"max":{"1min":55.738,"5min":55.738,"15min":401.80099999999999},"last":6.569},{"interface":"front","average":{"1min":18.251000000000001,"5min":11.83,"15min":15.614000000000001},"min":{"1min":6.1360000000000001,"5min":4.9820000000000002,"15min":4.7690000000000001},"max":{"1min":83.034000000000006,"5min":83.034000000000006,"15min":139.387},"last":22.186}]},{"osd":1,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":16.733000000000001,"5min":13.003,"15min":17.408000000000001},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":5.5540000000000003},"max":{"1min":85.837999999999994,"5min":85.837999999999994,"15min":168.71299999999999},"last":24.943000000000001},{"interface":"front","average":{"1min":24.350000000000001,"5min":17.399999999999999,"15min":23.219999999999999},"min":{"1min":11.083,"5min":10.079000000000001,"15min":10.079000000000001},"max":{"1min":87.222999999999999,"5min":87.222999999999999,"15min":297.87299999999999},"last":32.146000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:34 2023","interfaces":[{"interface":"back","average":{"1min":10.785,"5min":11.074999999999999,"15min":16.484999999999999},"min":{"1min":5.5460000000000003,"5min":5.5460000000000003,"15min":4.6980000000000004},"max":{"1min":17.513000000000002,"5min":20.484999999999999,"15min":222.297},"last":23.408000000000001},{"interface":"front","average":{"1min":16.869,"5min":12.349,"15min":17.806999999999999},"min":{"1min":7.9729999999999999,"5min":6.0209999999999999,"15min":5.7789999999999999},"max":{"1min":88.489000000000004,"5min":88.489000000000004,"15min":402.50200000000001},"last":24.07}]},{"osd":3,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.141999999999999,"5min":14.019,"15min":20.75},"min":{"1min":8,"5min":6.6630000000000003,"15min":6.5190000000000001},"max":{"1min":86.459000000000003,"5min":86.459000000000003,"15min":295.20499999999998},"last":33.392000000000003},{"interface":"front","average":{"1min":23.57,"5min":16.802,"15min":23.039999999999999},"min":{"1min":7.7939999999999996,"5min":7.7939999999999996,"15min":7.7939999999999996},"max":{"1min":95.340000000000003,"5min":95.340000000000003,"15min":154.19200000000001},"last":35.283999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":24.349,"5min":18.375,"15min":23.802},"min":{"1min":9.2729999999999997,"5min":9.2729999999999997,"15min":9.0760000000000005},"max":{"1min":84.105999999999995,"5min":84.105999999999995,"15min":337.97199999999998},"last":31.273},{"interface":"front","average":{"1min":20.994,"5min":15.461,"15min":19.864000000000001},"min":{"1min":11.005000000000001,"5min":9.0180000000000007,"15min":6.7350000000000003},"max":{"1min":67.284000000000006,"5min":67.284000000000006,"15min":169.846},"last":30.152999999999999}]},{"osd":6,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.216999999999999,"5min":17.192,"15min":19.756},"min":{"1min":9.3260000000000005,"5min":8.4109999999999996,"15min":7.4109999999999996},"max":{"1min":67.876999999999995,"5min":67.876999999999995,"15min":169.93199999999999},"last":30.965},{"interface":"front","average":{"1min":26.831,"5min":19.538,"15min":24.748999999999999},"min":{"1min":14.941000000000001,"5min":10.512,"15min":8.7810000000000006},"max":{"1min":89.180000000000007,"5min":89.180000000000007,"15min":395.5},"last":49.508000000000003}]},{"osd":7,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":25.442,"5min":14.970000000000001,"15min":23.666},"min":{"1min":13.343,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":81.359999999999999,"5min":81.359999999999999,"15min":417.26299999999998},"last":32.779000000000003},{"interface":"front","average":{"1min":20.204000000000001,"5min":17.381,"15min":21.201000000000001},"min":{"1min":7.5359999999999996,"5min":7.5359999999999996,"15min":6.5780000000000003},"max":{"1min":84.742000000000004,"5min":84.742000000000004,"15min":352.34100000000001},"last":26.891999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575232,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69436,"kb_used_data":6088,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302404,"statfs":{"total":96636764160,"available":96565661696,"internally_reserved":0,"allocated":6234112,"data_stored":2932452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.9109999999999996,"15min":15.105},"min":{"1min":4.9039999999999999,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":19.489999999999998,"5min":19.489999999999998,"15min":218.566},"last":9.8170000000000002},{"interface":"front","average":{"1min":9.6110000000000007,"5min":10.023,"15min":16.123999999999999},"min":{"1min":5.4039999999999999,"5min":5.1520000000000001,"15min":4.8300000000000001},"max":{"1min":19.678999999999998,"5min":19.678999999999998,"15min":299.358},"last":6.6920000000000002}]},{"osd":1,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":11.888999999999999,"15min":17.398},"min":{"1min":6.6440000000000001,"5min":6.0190000000000001,"15min":5.9249999999999998},"max":{"1min":16.940000000000001,"5min":18.978000000000002,"15min":291.13200000000001},"last":15.452},{"interface":"front","average":{"1min":11.286,"5min":10.930999999999999,"15min":14.776},"min":{"1min":5.3490000000000002,"5min":5.3490000000000002,"15min":4.9509999999999996},"max":{"1min":16.937000000000001,"5min":18.469000000000001,"15min":147.71000000000001},"last":7.9950000000000001}]},{"osd":2,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":11.776999999999999,"15min":17.225000000000001},"min":{"1min":7.2880000000000003,"5min":6.8819999999999997,"15min":6.1539999999999999},"max":{"1min":86.319000000000003,"5min":86.319000000000003,"15min":300.22399999999999},"last":8.9290000000000003},{"interface":"front","average":{"1min":15.244,"5min":14.737,"15min":20.122},"min":{"1min":11.385,"5min":8.7119999999999997,"15min":6.7009999999999996},"max":{"1min":26.940999999999999,"5min":26.940999999999999,"15min":258.58499999999998},"last":16.727}]},{"osd":3,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.311,"5min":14.824999999999999,"15min":22.536999999999999},"min":{"1min":9.6920000000000002,"5min":7.4279999999999999,"15min":6.4219999999999997},"max":{"1min":51.329999999999998,"5min":51.329999999999998,"15min":400.41699999999997},"last":17.338000000000001},{"interface":"front","average":{"1min":15.747999999999999,"5min":13.279,"15min":19.422000000000001},"min":{"1min":9.8249999999999993,"5min":7.0460000000000003,"15min":7.0460000000000003},"max":{"1min":49.639000000000003,"5min":49.639000000000003,"15min":273.839},"last":11.558}]},{"osd":5,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.038,"5min":15.727,"15min":22.605},"min":{"1min":10.035,"5min":7.4939999999999998,"15min":6.7569999999999997},"max":{"1min":33.755000000000003,"5min":33.755000000000003,"15min":397.96199999999999},"last":10.535},{"interface":"front","average":{"1min":17.222999999999999,"5min":14.757,"15min":20.649000000000001},"min":{"1min":10.176,"5min":7.0129999999999999,"15min":6.9900000000000002},"max":{"1min":59.944000000000003,"5min":59.944000000000003,"15min":291.952},"last":12.727}]},{"osd":6,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":19.376999999999999,"5min":15.308999999999999,"15min":20.122},"min":{"1min":8.5579999999999998,"5min":7.8940000000000001,"15min":6.1399999999999997},"max":{"1min":50.722999999999999,"5min":50.722999999999999,"15min":296.37},"last":16.100999999999999},{"interface":"front","average":{"1min":25.309000000000001,"5min":17.347000000000001,"15min":21.280000000000001},"min":{"1min":11.715,"5min":8.9130000000000003,"15min":7.8860000000000001},"max":{"1min":111.55,"5min":111.55,"15min":232.64099999999999},"last":12.116}]},{"osd":7,"last update":"Fri Dec 8 00:17:14 2023","interfaces":[{"interface":"back","average":{"1min":17.195,"5min":16.981999999999999,"15min":22.972999999999999},"min":{"1min":11.882,"5min":9.6539999999999999,"15min":7.4279999999999999},"max":{"1min":22.263999999999999,"5min":22.263999999999999,"15min":235.828},"last":13.333},{"interface":"front","average":{"1min":16.57,"5min":16.664000000000001,"15min":24.167000000000002},"min":{"1min":11.945,"5min":9.6530000000000005,"15min":9.0869999999999997},"max":{"1min":19.780000000000001,"5min":22.067,"15min":298.74599999999998},"last":14.007999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T00:18:13.462 INFO:tasks.ceph:pgid 15.1f last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.462 INFO:tasks.ceph:pgid 15.1e last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.462 INFO:tasks.ceph:pgid 15.1d last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.462 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.462 INFO:tasks.ceph:pgid 15.3 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.1 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.0 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.7 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.6 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.463 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 15.d last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 15.c last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 15.f last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.464 INFO:tasks.ceph:pgid 15.8 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 15.b last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 15.a last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-07T23:46:11.191893+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=46, tm_sec=11, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 15.18 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 15.19 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.465 INFO:tasks.ceph:pgid 15.1b last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.15 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.16 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.17 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.10 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.11 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.12 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:13.466 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-08T00:18:16.094 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:12.567 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.100 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:12.574 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.114 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:12.588 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.142 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:12.587 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.146 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:12.619 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.147 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:12.593 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.148 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:12.622 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.161 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:12.607 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:16.163 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:12.636 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.041 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:17.514 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.043 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:17.517 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.060 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:17.533 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.091 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:17.564 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.094 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:17.567 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.106 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:17.552 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.107 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:17.580 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.107 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:17.553 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:21.121 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:17.567 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.049 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:22.523 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.053 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:22.527 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.066 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:22.540 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.101 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:22.547 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.101 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:22.575 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.102 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:22.548 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.104 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:22.578 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.115 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:22.561 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:26.117 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:22.591 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:33.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-08T00:18:33.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.566+0000 7f994b590640 10 monclient: get_monmap_and_config 2023-12-08T00:18:33.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.566+0000 7f994b590640 10 monclient: build_initial_monmap 2023-12-08T00:18:33.575 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.566+0000 7f994b590640 10 monclient: monmap: 2023-12-08T00:18:33.575 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:33.575 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:33.576 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:33.567464+0000 2023-12-08T00:18:33.576 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:33.567464+0000 2023-12-08T00:18:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:33.578 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.566+0000 7f994b590640 1 Processor -- start 2023-12-08T00:18:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 1 -- start start 2023-12-08T00:18:33.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient: init 2023-12-08T00:18:33.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:33.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-12-08T00:18:33.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f994413d300 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:33.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d6e0 0x7f99440a8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:33.579 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient(hunting): picked mon.noname-b con 0x7f994413d6e0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:33.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9944072830 0x7f99441431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:33.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient(hunting): picked mon.noname-c con 0x7f9944072830 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:33.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99440a84a0 con 0x7f994413d300 2023-12-08T00:18:33.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:33.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99440aa2f0 con 0x7f9944072830 2023-12-08T00:18:33.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:33.580 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99440aa560 con 0x7f994413d6e0 2023-12-08T00:18:33.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:33.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.567+0000 7f994b590640 10 monclient(hunting): authenticate will time out at 3313.755706s 2023-12-08T00:18:33.581 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9948b2b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d6e0 0x7f99440a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:33.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9949b2d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9944072830 0x7f99441431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:33.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9948b2b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d6e0 0x7f99440a8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:35936/0 (socket says 172.21.15.129:35936) 2023-12-08T00:18:33.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9948b2b640 1 -- 172.21.15.129:0/3465720114 learned_addr learned my addr 172.21.15.129:0/3465720114 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:33.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9948b2b640 10 monclient(hunting): get_auth_request con 0x7f994413d6e0 auth_method 0 2023-12-08T00:18:33.582 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9948b2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:33.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9948b2b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:33.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.568+0000 7f9948b2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:33.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.569+0000 7f9949b2d640 10 monclient(hunting): get_auth_request con 0x7f9944072830 auth_method 0 2023-12-08T00:18:33.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.569+0000 7f9949b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:33.583 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.569+0000 7f9949b2d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:33.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.569+0000 7f9949b2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:33.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.569+0000 7f9949b2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:33.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.569+0000 7f9949b2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:33.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.569+0000 7f9949b2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:33.584 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.570+0000 7f9948b2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:33.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.570+0000 7f9948b2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:33.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.570+0000 7f9948b2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:33.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 10 monclient(hunting): handle_auth_done global_id 20314 payload 274 2023-12-08T00:18:33.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 1 -- 172.21.15.129:0/3465720114 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d6e0 msgr2=0x7f99440a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:33.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 1 --2- 172.21.15.129:0/3465720114 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d6e0 0x7f99440a8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:33.585 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 1 -- 172.21.15.129:0/3465720114 >> v1:172.21.15.184:6789/0 conn(0x7f994413d300 legacy=0x7f99440b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 10 monclient: _finish_hunting 0 2023-12-08T00:18:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 1 monclient: found mon.noname-c 2023-12-08T00:18:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.184:3300/0 2023-12-08T00:18:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.571+0000 7f9949b2d640 1 -- 172.21.15.129:0/3465720114 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944143700 con 0x7f9944072830 2023-12-08T00:18:33.591 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.573+0000 7f9949b2d640 1 --2- 172.21.15.129:0/3465720114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 0x7f99441431c0 secure :-1 s=READY pgs=4884 cs=0 l=1 rev1=1 crypto rx=0x7f99400097a0 tx=0x7f994002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1be721e0d7264c50 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 1 -- 172.21.15.129:0/3465720114 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994002e070 con 0x7f9944072830 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: dump: 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:33.592 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: _finish_auth 0 2023-12-08T00:18:33.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: _check_auth_tickets 2023-12-08T00:18:33.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:33.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 1 -- 172.21.15.129:0/3465720114 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99400315a0 con 0x7f9944072830 2023-12-08T00:18:33.594 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:33.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 1 -- 172.21.15.129:0/3465720114 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940031930 con 0x7f9944072830 2023-12-08T00:18:33.595 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 5 monclient: authenticate success, global_id 20314 2023-12-08T00:18:33.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:33.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 4 set_mon_vals no callback set 2023-12-08T00:18:33.597 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:33.598 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-08T00:18:33.598 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f993bfff640 10 monclient: dump: 2023-12-08T00:18:33.598 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 10 monclient: shutdown 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 1 -- 172.21.15.129:0/3465720114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 msgr2=0x7f99441431c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 1 --2- 172.21.15.129:0/3465720114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 0x7f99441431c0 secure :-1 s=READY pgs=4884 cs=0 l=1 rev1=1 crypto rx=0x7f99400097a0 tx=0x7f994002b840 comp rx=0 tx=0).stop 2023-12-08T00:18:33.599 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 1 -- 172.21.15.129:0/3465720114 shutdown_connections 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 1 --2- 172.21.15.129:0/3465720114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 0x7f99441431c0 unknown :-1 s=CLOSED pgs=4884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 1 --2- 172.21.15.129:0/3465720114 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d6e0 0x7f99440a8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.578+0000 7f994b590640 1 -- 172.21.15.129:0/3465720114 >> 172.21.15.129:0/3465720114 conn(0x7f994406ac00 msgr2=0x7f99440b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.579+0000 7f994b590640 1 -- 172.21.15.129:0/3465720114 shutdown_connections 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.579+0000 7f994b590640 1 -- 172.21.15.129:0/3465720114 wait complete. 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.579+0000 7f994b590640 10 monclient: build_initial_monmap 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.579+0000 7f994b590640 10 monclient: monmap: 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:33.600 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.579+0000 7f994b590640 1 librados: starting msgr at 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.579+0000 7f994b590640 1 librados: starting objecter 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 Processor -- start 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 -- start start 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 librados: setting wanted keys 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 librados: calling monclient init 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient: init 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 0x7f9944070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): picked mon.noname-b con 0x7f9944072830 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d300 0x7f99440aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): picked mon.noname-c con 0x7f994413d300 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f994413d6e0 0x7f99440ab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): picked mon.noname-a con 0x7f994413d6e0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9944138460 con 0x7f994413d6e0 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99440ab630 con 0x7f9944072830 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9944149c50 con 0x7f994413d300 2023-12-08T00:18:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.580+0000 7f994b590640 10 monclient(hunting): authenticate will time out at 3313.768601s 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f994413d6e0 0x7f99440ab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d300 0x7f99440aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f994413d6e0 0x7f99440ab0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57186/0 (socket says 172.21.15.129:57186) 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 1 -- 172.21.15.129:0/548236208 learned_addr learned my addr 172.21.15.129:0/548236208 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 10 monclient(hunting): get_auth_request con 0x7f994413d6e0 auth_method 0 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:33.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:33.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 10 monclient(hunting): get_auth_request con 0x7f994413d300 auth_method 0 2023-12-08T00:18:33.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:33.604 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:33.605 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:33.605 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f994932c640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 0x7f9944070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:33.605 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:33.605 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:33.605 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9949b2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:33.606 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f994932c640 10 monclient(hunting): get_auth_request con 0x7f9944072830 auth_method 0 2023-12-08T00:18:33.606 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f994932c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:33.606 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f994932c640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:33.606 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f994932c640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:33.606 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:33.606 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:33.606 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.581+0000 7f9948b2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.582+0000 7f994932c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.582+0000 7f994932c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.582+0000 7f994932c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 10 monclient(hunting): handle_auth_done global_id 18443 payload 824 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 1 -- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 msgr2=0x7f9944070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 0x7f9944070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 1 -- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f994413d6e0 msgr2=0x7f99440ab0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f994413d6e0 0x7f99440ab0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 10 monclient: _finish_hunting 0 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 1 monclient: found mon.noname-c 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.129:3301/0 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9948b2b640 1 -- 172.21.15.129:0/548236208 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9940009400 con 0x7f994413d300 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.584+0000 7f9949b2d640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f994413d6e0 0x7f99440ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.587+0000 7f9948b2b640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d300 0x7f99440aabb0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7f993400e870 tx=0x7f993400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:33.607 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.587+0000 7f9939ffb640 10 client.?.objecter ms_handle_connect 0x7f994413d300 2023-12-08T00:18:33.608 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.587+0000 7f9939ffb640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 1 -- 172.21.15.129:0/548236208 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993400ee60 con 0x7f994413d300 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: dump: 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:33.609 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: _finish_auth 0 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: _check_auth_tickets 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 1 -- 172.21.15.129:0/548236208 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993400c940 con 0x7f994413d300 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 5 monclient: authenticate success, global_id 18443 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 10 monclient: _renew_subs 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99440ac6c0 con 0x7f994413d300 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 1 -- 172.21.15.129:0/548236208 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993400ccd0 con 0x7f994413d300 2023-12-08T00:18:33.610 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f9939ffb640 10 monclient: dump: 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f993b7fe640 4 set_mon_vals no callback set 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 10 client.18443.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 10 monclient: _renew_subs 2023-12-08T00:18:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:33.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9944149f00 con 0x7f994413d300 2023-12-08T00:18:33.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.590+0000 7f994b590640 1 librados: init done 2023-12-08T00:18:33.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.591+0000 7f994b590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:33.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.592+0000 7f994b590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:33.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.592+0000 7f994b590640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.129:3301/0 2023-12-08T00:18:33.612 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.592+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994404e840 con 0x7f994413d300 2023-12-08T00:18:33.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.863+0000 7f9939ffb640 1 -- 172.21.15.129:0/548236208 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9934013470 con 0x7f994413d300 2023-12-08T00:18:33.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.863+0000 7f9939ffb640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f992806c820 0x7f992806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:33.866 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.864+0000 7f994932c640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f992806c820 0x7f992806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:33.867 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.864+0000 7f994932c640 10 monclient: get_auth_request con 0x7f992806c820 auth_method 0 2023-12-08T00:18:33.867 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.864+0000 7f9939ffb640 10 client.18443.objecter ms_handle_connect 0x7f992806c820 2023-12-08T00:18:33.867 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.864+0000 7f994932c640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f992806c820 0x7f992806ece0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f993c001f10 tx=0x7f993c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 1 -- 172.21.15.129:0/548236208 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(1049..1049 src has 999..1049) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f993400ce30 con 0x7f994413d300 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 10 client.18443.objecter ms_dispatch 0x7f9944000b90 osd_map(1049..1049 src has 999..1049) v4 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 3 client.18443.objecter handle_osd_map got epochs [1049,1049] > 0 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 3 client.18443.objecter handle_osd_map decoding full epoch 1049 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 20 client.18443.objecter dump_active .. 0 homeless 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 1 -- 172.21.15.129:0/548236208 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9934054ef0 con 0x7f994413d300 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:33.875+0000 7f9939ffb640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:34.044 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.042+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9944142e00 con 0x7f992806c820 2023-12-08T00:18:34.049 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.047+0000 7f9939ffb640 1 -- 172.21.15.129:0/548236208 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151844 (secure 0 0 0) 0x7f9944142e00 con 0x7f992806c820 2023-12-08T00:18:34.051 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-08T00:18:34.052 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-08T00:18:34.053 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.049+0000 7f994b590640 10 librados: watch_flush enter 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.049+0000 7f994b590640 10 librados: watch_flush exit 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.049+0000 7f994b590640 20 client.18443.objecter shutdown clearing up homeless session... 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.049+0000 7f994b590640 10 client.18443.objecter successfully canceled tick 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f992806c820 msgr2=0x7f992806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f992806c820 0x7f992806ece0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f993c001f10 tx=0x7f993c009040 comp rx=0 tx=0).stop 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 10 monclient: shutdown 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d300 msgr2=0x7f99440aabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d300 0x7f99440aabb0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7f993400e870 tx=0x7f993400ed30 comp rx=0 tx=0).stop 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 shutdown_connections 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f992806c820 0x7f992806ece0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f994413d6e0 0x7f99440ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:34.054 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f994413d300 0x7f99440aabb0 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:34.055 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 --2- 172.21.15.129:0/548236208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9944072830 0x7f9944070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:34.055 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 >> 172.21.15.129:0/548236208 conn(0x7f994406ac00 msgr2=0x7f99440b29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:34.055 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 shutdown_connections 2023-12-08T00:18:34.055 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 -- 172.21.15.129:0/548236208 wait complete. 2023-12-08T00:18:34.055 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:34.050+0000 7f994b590640 1 librados: shutdown 2023-12-08T00:18:34.063 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":2028,"stamp":"2023-12-08T00:18:32.141582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":97,"num_write":134,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":583476,"kb_used_data":45300,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754391244,"statfs":{"total":773094113280,"available":772496633856,"internally_reserved":0,"allocated":46387200,"data_stored":21850546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002458"},"pg_stats":[{"pgid":"15.1f","version":"0'0","reported_seq":1014,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.928334+0000","last_change":"2023-12-08T00:18:18.928334+0000","last_active":"2023-12-08T00:18:18.928334+0000","last_peered":"2023-12-08T00:18:18.928334+0000","last_clean":"2023-12-08T00:18:18.928334+0000","last_became_active":"2023-12-07T23:47:23.187382+0000","last_became_peered":"2023-12-07T23:47:23.187382+0000","last_unstale":"2023-12-08T00:18:18.928334+0000","last_undegraded":"2023-12-08T00:18:18.928334+0000","last_fullsized":"2023-12-08T00:18:18.928334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.926408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.926408+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.926408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:43:17.526124+0000","scrub_duration":0.071049899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.016342+0000","last_change":"2023-12-08T00:18:21.017002+0000","last_active":"2023-12-08T00:18:21.016342+0000","last_peered":"2023-12-08T00:18:21.016342+0000","last_clean":"2023-12-08T00:18:21.016342+0000","last_became_active":"2023-12-07T23:47:26.088721+0000","last_became_peered":"2023-12-07T23:47:26.088721+0000","last_unstale":"2023-12-08T00:18:21.016342+0000","last_undegraded":"2023-12-08T00:18:21.016342+0000","last_fullsized":"2023-12-08T00:18:21.016342+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.014984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.014984+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.014984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T12:16:56.647346+0000","scrub_duration":0.02649731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1025,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.568533+0000","last_change":"2023-12-08T00:18:10.233299+0000","last_active":"2023-12-08T00:18:11.568533+0000","last_peered":"2023-12-08T00:18:11.568533+0000","last_clean":"2023-12-08T00:18:11.568533+0000","last_became_active":"2023-12-07T23:47:18.658074+0000","last_became_peered":"2023-12-07T23:47:18.658074+0000","last_unstale":"2023-12-08T00:18:11.568533+0000","last_undegraded":"2023-12-08T00:18:11.568533+0000","last_fullsized":"2023-12-08T00:18:11.568533+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:10.231235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:10.231235+0000","last_clean_scrub_stamp":"2023-12-08T00:18:10.231235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:33:04.158595+0000","scrub_duration":0.060972367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1000,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:15.211477+0000","last_change":"2023-12-08T00:18:15.211477+0000","last_active":"2023-12-08T00:18:15.211477+0000","last_peered":"2023-12-08T00:18:15.211477+0000","last_clean":"2023-12-08T00:18:15.211477+0000","last_became_active":"2023-12-07T23:47:39.133797+0000","last_became_peered":"2023-12-07T23:47:39.133797+0000","last_unstale":"2023-12-08T00:18:15.211477+0000","last_undegraded":"2023-12-08T00:18:15.211477+0000","last_fullsized":"2023-12-08T00:18:15.211477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:15.209401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:15.209401+0000","last_clean_scrub_stamp":"2023-12-08T00:18:15.209401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:18:52.612018+0000","scrub_duration":0.040096010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":998,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.036392+0000","last_change":"2023-12-07T23:47:20.151369+0000","last_active":"2023-12-08T00:18:11.036392+0000","last_peered":"2023-12-08T00:18:11.036392+0000","last_clean":"2023-12-08T00:18:11.036392+0000","last_became_active":"2023-12-07T23:47:20.132690+0000","last_became_peered":"2023-12-07T23:47:20.132690+0000","last_unstale":"2023-12-08T00:18:11.036392+0000","last_undegraded":"2023-12-08T00:18:11.036392+0000","last_fullsized":"2023-12-08T00:18:11.036392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.267961+0000","last_change":"2023-12-08T00:18:14.984114+0000","last_active":"2023-12-08T00:18:16.267961+0000","last_peered":"2023-12-08T00:18:16.267961+0000","last_clean":"2023-12-08T00:18:16.267961+0000","last_became_active":"2023-12-07T23:47:18.926939+0000","last_became_peered":"2023-12-07T23:47:18.926939+0000","last_unstale":"2023-12-08T00:18:16.267961+0000","last_undegraded":"2023-12-08T00:18:16.267961+0000","last_fullsized":"2023-12-08T00:18:16.267961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:14.982044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:14.982044+0000","last_clean_scrub_stamp":"2023-12-08T00:18:14.982044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:53:58.858191+0000","scrub_duration":0.029090370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1","version":"1048'8","reported_seq":1021,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.206631+0000","last_change":"2023-12-08T00:18:14.899000+0000","last_active":"2023-12-08T00:18:16.206631+0000","last_peered":"2023-12-08T00:18:16.206631+0000","last_clean":"2023-12-08T00:18:16.206631+0000","last_became_active":"2023-12-07T23:47:15.386185+0000","last_became_peered":"2023-12-07T23:47:15.386185+0000","last_unstale":"2023-12-08T00:18:16.206631+0000","last_undegraded":"2023-12-08T00:18:16.206631+0000","last_fullsized":"2023-12-08T00:18:16.206631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'8","last_scrub_stamp":"2023-12-08T00:18:14.896730+0000","last_deep_scrub":"1048'8","last_deep_scrub_stamp":"2023-12-08T00:18:14.896730+0000","last_clean_scrub_stamp":"2023-12-08T00:18:14.896730+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:26:28.033722+0000","scrub_duration":0.071940854999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.991668+0000","last_change":"2023-12-08T00:18:18.992364+0000","last_active":"2023-12-08T00:18:18.991668+0000","last_peered":"2023-12-08T00:18:18.991668+0000","last_clean":"2023-12-08T00:18:18.991668+0000","last_became_active":"2023-12-07T23:47:18.700591+0000","last_became_peered":"2023-12-07T23:47:18.700591+0000","last_unstale":"2023-12-08T00:18:18.991668+0000","last_undegraded":"2023-12-08T00:18:18.991668+0000","last_fullsized":"2023-12-08T00:18:18.991668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.990338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.990338+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.990338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:58:57.319194+0000","scrub_duration":0.025705731999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:23.067992+0000","last_change":"2023-12-08T00:18:23.069096+0000","last_active":"2023-12-08T00:18:23.067992+0000","last_peered":"2023-12-08T00:18:23.067992+0000","last_clean":"2023-12-08T00:18:23.067992+0000","last_became_active":"2023-12-07T23:47:16.101044+0000","last_became_peered":"2023-12-07T23:47:16.101044+0000","last_unstale":"2023-12-08T00:18:23.067992+0000","last_undegraded":"2023-12-08T00:18:23.067992+0000","last_fullsized":"2023-12-08T00:18:23.067992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:23.063901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:23.063901+0000","last_clean_scrub_stamp":"2023-12-08T00:18:23.063901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:07:11.581982+0000","scrub_duration":0.072925316000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:22.000467+0000","last_change":"2023-12-08T00:18:22.001151+0000","last_active":"2023-12-08T00:18:22.000467+0000","last_peered":"2023-12-08T00:18:22.000467+0000","last_clean":"2023-12-08T00:18:22.000467+0000","last_became_active":"2023-12-07T23:47:18.123992+0000","last_became_peered":"2023-12-07T23:47:18.123992+0000","last_unstale":"2023-12-08T00:18:22.000467+0000","last_undegraded":"2023-12-08T00:18:22.000467+0000","last_fullsized":"2023-12-08T00:18:22.000467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.999077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.999077+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.999077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T00:20:22.858450+0000","scrub_duration":0.039753019000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":998,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.052413+0000","last_change":"2023-12-07T23:47:19.649219+0000","last_active":"2023-12-08T00:18:11.052413+0000","last_peered":"2023-12-08T00:18:11.052413+0000","last_clean":"2023-12-08T00:18:11.052413+0000","last_became_active":"2023-12-07T23:47:19.507339+0000","last_became_peered":"2023-12-07T23:47:19.507339+0000","last_unstale":"2023-12-08T00:18:11.052413+0000","last_undegraded":"2023-12-08T00:18:11.052413+0000","last_fullsized":"2023-12-08T00:18:11.052413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1022,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.825211+0000","last_change":"2023-12-08T00:18:24.825887+0000","last_active":"2023-12-08T00:18:24.825211+0000","last_peered":"2023-12-08T00:18:24.825211+0000","last_clean":"2023-12-08T00:18:24.825211+0000","last_became_active":"2023-12-07T23:48:06.627236+0000","last_became_peered":"2023-12-07T23:48:06.627236+0000","last_unstale":"2023-12-08T00:18:24.825211+0000","last_undegraded":"2023-12-08T00:18:24.825211+0000","last_fullsized":"2023-12-08T00:18:24.825211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.823891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.823891+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.823891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:14:41.955238+0000","scrub_duration":0.037228697999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1013,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.027356+0000","last_change":"2023-12-07T23:47:18.049591+0000","last_active":"2023-12-08T00:18:11.027356+0000","last_peered":"2023-12-08T00:18:11.027356+0000","last_clean":"2023-12-08T00:18:11.027356+0000","last_became_active":"2023-12-07T23:47:18.028936+0000","last_became_peered":"2023-12-07T23:47:18.028936+0000","last_unstale":"2023-12-08T00:18:11.027356+0000","last_undegraded":"2023-12-08T00:18:11.027356+0000","last_fullsized":"2023-12-08T00:18:11.027356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"1047'90","reported_seq":1135,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.199266+0000","last_change":"2023-12-08T00:18:10.088999+0000","last_active":"2023-12-08T00:18:16.199266+0000","last_peered":"2023-12-08T00:18:16.199266+0000","last_clean":"2023-12-08T00:18:16.199266+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-08T00:18:16.199266+0000","last_undegraded":"2023-12-08T00:18:16.199266+0000","last_fullsized":"2023-12-08T00:18:16.199266+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"1047'90","last_scrub_stamp":"2023-12-08T00:18:10.084655+0000","last_deep_scrub":"1047'90","last_deep_scrub_stamp":"2023-12-08T00:18:10.084655+0000","last_clean_scrub_stamp":"2023-12-08T00:18:10.084655+0000","objects_scrubbed":2,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:51:22.482725+0000","scrub_duration":0.204482201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1011,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:19.754747+0000","last_change":"2023-12-08T00:18:19.755399+0000","last_active":"2023-12-08T00:18:19.754747+0000","last_peered":"2023-12-08T00:18:19.754747+0000","last_clean":"2023-12-08T00:18:19.754747+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-08T00:18:19.754747+0000","last_undegraded":"2023-12-08T00:18:19.754747+0000","last_fullsized":"2023-12-08T00:18:19.754747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:19.752924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:19.752924+0000","last_clean_scrub_stamp":"2023-12-08T00:18:19.752924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:35:07.286744+0000","scrub_duration":0.028125618000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.839346+0000","last_change":"2023-12-08T00:18:21.840013+0000","last_active":"2023-12-08T00:18:21.839346+0000","last_peered":"2023-12-08T00:18:21.839346+0000","last_clean":"2023-12-08T00:18:21.839346+0000","last_became_active":"2023-12-07T23:47:18.104193+0000","last_became_peered":"2023-12-07T23:47:18.104193+0000","last_unstale":"2023-12-08T00:18:21.839346+0000","last_undegraded":"2023-12-08T00:18:21.839346+0000","last_fullsized":"2023-12-08T00:18:21.839346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.837952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.837952+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.837952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:29:43.938898+0000","scrub_duration":0.057958458999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1012,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.711660+0000","last_change":"2023-12-08T00:18:15.661804+0000","last_active":"2023-12-08T00:18:16.711660+0000","last_peered":"2023-12-08T00:18:16.711660+0000","last_clean":"2023-12-08T00:18:16.711660+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-08T00:18:16.711660+0000","last_undegraded":"2023-12-08T00:18:16.711660+0000","last_fullsized":"2023-12-08T00:18:16.711660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:15.659834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:15.659834+0000","last_clean_scrub_stamp":"2023-12-08T00:18:15.659834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:23:11.986021+0000","scrub_duration":0.02440811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1019,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:20.275155+0000","last_change":"2023-12-08T00:18:20.275795+0000","last_active":"2023-12-08T00:18:20.275155+0000","last_peered":"2023-12-08T00:18:20.275155+0000","last_clean":"2023-12-08T00:18:20.275155+0000","last_became_active":"2023-12-07T23:47:25.699009+0000","last_became_peered":"2023-12-07T23:47:25.699009+0000","last_unstale":"2023-12-08T00:18:20.275155+0000","last_undegraded":"2023-12-08T00:18:20.275155+0000","last_fullsized":"2023-12-08T00:18:20.275155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:20.273808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:20.273808+0000","last_clean_scrub_stamp":"2023-12-08T00:18:20.273808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:21:30.745491+0000","scrub_duration":0.022630939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1017,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.950346+0000","last_change":"2023-12-08T00:18:12.950346+0000","last_active":"2023-12-08T00:18:12.950346+0000","last_peered":"2023-12-08T00:18:12.950346+0000","last_clean":"2023-12-08T00:18:12.950346+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-08T00:18:12.950346+0000","last_undegraded":"2023-12-08T00:18:12.950346+0000","last_fullsized":"2023-12-08T00:18:12.950346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.947112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.947112+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.947112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:18:48.425554+0000","scrub_duration":0.031302554000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1018,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.283840+0000","last_change":"2023-12-08T00:18:21.284555+0000","last_active":"2023-12-08T00:18:21.283840+0000","last_peered":"2023-12-08T00:18:21.283840+0000","last_clean":"2023-12-08T00:18:21.283840+0000","last_became_active":"2023-12-07T23:47:26.173821+0000","last_became_peered":"2023-12-07T23:47:26.173821+0000","last_unstale":"2023-12-08T00:18:21.283840+0000","last_undegraded":"2023-12-08T00:18:21.283840+0000","last_fullsized":"2023-12-08T00:18:21.283840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.282454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.282454+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.282454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:51:49.950914+0000","scrub_duration":0.068318492999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1011,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.564346+0000","last_change":"2023-12-08T00:18:11.564346+0000","last_active":"2023-12-08T00:18:11.564346+0000","last_peered":"2023-12-08T00:18:11.564346+0000","last_clean":"2023-12-08T00:18:11.564346+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-08T00:18:11.564346+0000","last_undegraded":"2023-12-08T00:18:11.564346+0000","last_fullsized":"2023-12-08T00:18:11.564346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:11.560752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:11.560752+0000","last_clean_scrub_stamp":"2023-12-08T00:18:11.560752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:53:55.150564+0000","scrub_duration":0.046001793999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1020,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.354981+0000","last_change":"2023-12-08T00:18:12.354981+0000","last_active":"2023-12-08T00:18:12.354981+0000","last_peered":"2023-12-08T00:18:12.354981+0000","last_clean":"2023-12-08T00:18:12.354981+0000","last_became_active":"2023-12-07T23:47:46.295177+0000","last_became_peered":"2023-12-07T23:47:46.295177+0000","last_unstale":"2023-12-08T00:18:12.354981+0000","last_undegraded":"2023-12-08T00:18:12.354981+0000","last_fullsized":"2023-12-08T00:18:12.354981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.352990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.352990+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.352990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:12:12.666032+0000","scrub_duration":0.042395521999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1009,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.014895+0000","last_change":"2023-12-08T00:18:10.951427+0000","last_active":"2023-12-08T00:18:12.014895+0000","last_peered":"2023-12-08T00:18:12.014895+0000","last_clean":"2023-12-08T00:18:12.014895+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-08T00:18:12.014895+0000","last_undegraded":"2023-12-08T00:18:12.014895+0000","last_fullsized":"2023-12-08T00:18:12.014895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:10.948269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:10.948269+0000","last_clean_scrub_stamp":"2023-12-08T00:18:10.948269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:22:31.595790+0000","scrub_duration":0.025839298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1002,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.231017+0000","last_change":"2023-12-08T00:18:12.231017+0000","last_active":"2023-12-08T00:18:12.231017+0000","last_peered":"2023-12-08T00:18:12.231017+0000","last_clean":"2023-12-08T00:18:12.231017+0000","last_became_active":"2023-12-07T23:47:18.206551+0000","last_became_peered":"2023-12-07T23:47:18.206551+0000","last_unstale":"2023-12-08T00:18:12.231017+0000","last_undegraded":"2023-12-08T00:18:12.231017+0000","last_fullsized":"2023-12-08T00:18:12.231017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.228656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.228656+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.228656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:03:03.814363+0000","scrub_duration":0.029704253999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1011,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:20.789672+0000","last_change":"2023-12-08T00:18:20.790324+0000","last_active":"2023-12-08T00:18:20.789672+0000","last_peered":"2023-12-08T00:18:20.789672+0000","last_clean":"2023-12-08T00:18:20.789672+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-08T00:18:20.789672+0000","last_undegraded":"2023-12-08T00:18:20.789672+0000","last_fullsized":"2023-12-08T00:18:20.789672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:20.788355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:20.788355+0000","last_clean_scrub_stamp":"2023-12-08T00:18:20.788355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:28:10.740579+0000","scrub_duration":0.026280008000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1003,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.061409+0000","last_change":"2023-12-08T00:18:16.061409+0000","last_active":"2023-12-08T00:18:16.061409+0000","last_peered":"2023-12-08T00:18:16.061409+0000","last_clean":"2023-12-08T00:18:16.061409+0000","last_became_active":"2023-12-07T23:47:15.770063+0000","last_became_peered":"2023-12-07T23:47:15.770063+0000","last_unstale":"2023-12-08T00:18:16.061409+0000","last_undegraded":"2023-12-08T00:18:16.061409+0000","last_fullsized":"2023-12-08T00:18:16.061409+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:16.059310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:16.059310+0000","last_clean_scrub_stamp":"2023-12-08T00:18:16.059310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:47:14.094698+0000","scrub_duration":0.10337629299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1027,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.844439+0000","last_change":"2023-12-08T00:18:12.844439+0000","last_active":"2023-12-08T00:18:12.844439+0000","last_peered":"2023-12-08T00:18:12.844439+0000","last_clean":"2023-12-08T00:18:12.844439+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-08T00:18:12.844439+0000","last_undegraded":"2023-12-08T00:18:12.844439+0000","last_fullsized":"2023-12-08T00:18:12.844439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.842337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.842337+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.842337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:19:54.361371+0000","scrub_duration":0.032484658999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1004,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.016563+0000","last_change":"2023-12-08T00:18:11.987531+0000","last_active":"2023-12-08T00:18:12.016563+0000","last_peered":"2023-12-08T00:18:12.016563+0000","last_clean":"2023-12-08T00:18:12.016563+0000","last_became_active":"2023-12-07T23:47:15.693158+0000","last_became_peered":"2023-12-07T23:47:15.693158+0000","last_unstale":"2023-12-08T00:18:12.016563+0000","last_undegraded":"2023-12-08T00:18:12.016563+0000","last_fullsized":"2023-12-08T00:18:12.016563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:11.985590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:11.985590+0000","last_clean_scrub_stamp":"2023-12-08T00:18:11.985590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:12:50.655676+0000","scrub_duration":0.041049627999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1010,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.799947+0000","last_change":"2023-12-08T00:18:24.800605+0000","last_active":"2023-12-08T00:18:24.799947+0000","last_peered":"2023-12-08T00:18:24.799947+0000","last_clean":"2023-12-08T00:18:24.799947+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-08T00:18:24.799947+0000","last_undegraded":"2023-12-08T00:18:24.799947+0000","last_fullsized":"2023-12-08T00:18:24.799947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.795744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.795744+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.795744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:11:59.912669+0000","scrub_duration":0.036985285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1011,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:13.945767+0000","last_change":"2023-12-08T00:18:13.945767+0000","last_active":"2023-12-08T00:18:13.945767+0000","last_peered":"2023-12-08T00:18:13.945767+0000","last_clean":"2023-12-08T00:18:13.945767+0000","last_became_active":"2023-12-07T23:47:21.229799+0000","last_became_peered":"2023-12-07T23:47:21.229799+0000","last_unstale":"2023-12-08T00:18:13.945767+0000","last_undegraded":"2023-12-08T00:18:13.945767+0000","last_fullsized":"2023-12-08T00:18:13.945767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:13.943649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:13.943649+0000","last_clean_scrub_stamp":"2023-12-08T00:18:13.943649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:13:49.022488+0000","scrub_duration":0.047138658999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.772914+0000","last_change":"2023-12-08T00:18:18.773561+0000","last_active":"2023-12-08T00:18:18.772914+0000","last_peered":"2023-12-08T00:18:18.772914+0000","last_clean":"2023-12-08T00:18:18.772914+0000","last_became_active":"2023-12-07T23:47:36.616695+0000","last_became_peered":"2023-12-07T23:47:36.616695+0000","last_unstale":"2023-12-08T00:18:18.772914+0000","last_undegraded":"2023-12-08T00:18:18.772914+0000","last_fullsized":"2023-12-08T00:18:18.772914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.771581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.771581+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.771581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:13:54.973664+0000","scrub_duration":0.042405533000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.796315+0000","last_change":"2023-12-08T00:18:24.797325+0000","last_active":"2023-12-08T00:18:24.796315+0000","last_peered":"2023-12-08T00:18:24.796315+0000","last_clean":"2023-12-08T00:18:24.796315+0000","last_became_active":"2023-12-07T23:47:36.567130+0000","last_became_peered":"2023-12-07T23:47:36.567130+0000","last_unstale":"2023-12-08T00:18:24.796315+0000","last_undegraded":"2023-12-08T00:18:24.796315+0000","last_fullsized":"2023-12-08T00:18:24.796315+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.794205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.794205+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.794205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:56:08.870545+0000","scrub_duration":0.025459683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:27.774320+0000","last_change":"2023-12-08T00:18:27.774978+0000","last_active":"2023-12-08T00:18:27.774320+0000","last_peered":"2023-12-08T00:18:27.774320+0000","last_clean":"2023-12-08T00:18:27.774320+0000","last_became_active":"2023-12-07T23:47:37.279080+0000","last_became_peered":"2023-12-07T23:47:37.279080+0000","last_unstale":"2023-12-08T00:18:27.774320+0000","last_undegraded":"2023-12-08T00:18:27.774320+0000","last_fullsized":"2023-12-08T00:18:27.774320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:27.772954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:27.772954+0000","last_clean_scrub_stamp":"2023-12-08T00:18:27.772954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:35:04.612518+0000","scrub_duration":0.078081676000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":997,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.046325+0000","last_change":"2023-12-07T23:47:21.394853+0000","last_active":"2023-12-08T00:18:11.046325+0000","last_peered":"2023-12-08T00:18:11.046325+0000","last_clean":"2023-12-08T00:18:11.046325+0000","last_became_active":"2023-12-07T23:47:21.389382+0000","last_became_peered":"2023-12-07T23:47:21.389382+0000","last_unstale":"2023-12-08T00:18:11.046325+0000","last_undegraded":"2023-12-08T00:18:11.046325+0000","last_fullsized":"2023-12-08T00:18:11.046325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T23:47:10.890608+0000","last_clean_scrub_stamp":"2023-12-07T23:47:10.890608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1018,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.193215+0000","last_change":"2023-12-08T00:18:24.193882+0000","last_active":"2023-12-08T00:18:24.193215+0000","last_peered":"2023-12-08T00:18:24.193215+0000","last_clean":"2023-12-08T00:18:24.193215+0000","last_became_active":"2023-12-07T23:47:20.968485+0000","last_became_peered":"2023-12-07T23:47:20.968485+0000","last_unstale":"2023-12-08T00:18:24.193215+0000","last_undegraded":"2023-12-08T00:18:24.193215+0000","last_fullsized":"2023-12-08T00:18:24.193215+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.191804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.191804+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.191804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:52:20.846742+0000","scrub_duration":0.047630634999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1000,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:17.190393+0000","last_change":"2023-12-08T00:18:17.191083+0000","last_active":"2023-12-08T00:18:17.190393+0000","last_peered":"2023-12-08T00:18:17.190393+0000","last_clean":"2023-12-08T00:18:17.190393+0000","last_became_active":"2023-12-07T23:47:21.395304+0000","last_became_peered":"2023-12-07T23:47:21.395304+0000","last_unstale":"2023-12-08T00:18:17.190393+0000","last_undegraded":"2023-12-08T00:18:17.190393+0000","last_fullsized":"2023-12-08T00:18:17.190393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:17.185405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:17.185405+0000","last_clean_scrub_stamp":"2023-12-08T00:18:17.185405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:07:03.933546+0000","scrub_duration":0.042693994999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1017,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:23.249331+0000","last_change":"2023-12-08T00:18:23.249975+0000","last_active":"2023-12-08T00:18:23.249331+0000","last_peered":"2023-12-08T00:18:23.249331+0000","last_clean":"2023-12-08T00:18:23.249331+0000","last_became_active":"2023-12-07T23:47:38.790323+0000","last_became_peered":"2023-12-07T23:47:38.790323+0000","last_unstale":"2023-12-08T00:18:23.249331+0000","last_undegraded":"2023-12-08T00:18:23.249331+0000","last_fullsized":"2023-12-08T00:18:23.249331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:23.248033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:23.248033+0000","last_clean_scrub_stamp":"2023-12-08T00:18:23.248033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:26:09.453050+0000","scrub_duration":0.072562849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:26.742425+0000","last_change":"2023-12-08T00:18:26.743113+0000","last_active":"2023-12-08T00:18:26.742425+0000","last_peered":"2023-12-08T00:18:26.742425+0000","last_clean":"2023-12-08T00:18:26.742425+0000","last_became_active":"2023-12-07T23:47:37.180489+0000","last_became_peered":"2023-12-07T23:47:37.180489+0000","last_unstale":"2023-12-08T00:18:26.742425+0000","last_undegraded":"2023-12-08T00:18:26.742425+0000","last_fullsized":"2023-12-08T00:18:26.742425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:26.741029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:26.741029+0000","last_clean_scrub_stamp":"2023-12-08T00:18:26.741029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:52:01.190729+0000","scrub_duration":0.025969953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1021,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.812103+0000","last_change":"2023-12-08T00:18:16.812103+0000","last_active":"2023-12-08T00:18:16.812103+0000","last_peered":"2023-12-08T00:18:16.812103+0000","last_clean":"2023-12-08T00:18:16.812103+0000","last_became_active":"2023-12-07T23:48:09.186728+0000","last_became_peered":"2023-12-07T23:48:09.186728+0000","last_unstale":"2023-12-08T00:18:16.812103+0000","last_undegraded":"2023-12-08T00:18:16.812103+0000","last_fullsized":"2023-12-08T00:18:16.812103+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:16.809929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:16.809929+0000","last_clean_scrub_stamp":"2023-12-08T00:18:16.809929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:14:52.287877+0000","scrub_duration":0.044075022999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1001,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.142254+0000","last_change":"2023-12-08T00:18:18.142932+0000","last_active":"2023-12-08T00:18:18.142254+0000","last_peered":"2023-12-08T00:18:18.142254+0000","last_clean":"2023-12-08T00:18:18.142254+0000","last_became_active":"2023-12-07T23:47:20.123312+0000","last_became_peered":"2023-12-07T23:47:20.123312+0000","last_unstale":"2023-12-08T00:18:18.142254+0000","last_undegraded":"2023-12-08T00:18:18.142254+0000","last_fullsized":"2023-12-08T00:18:18.142254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.140805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.140805+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.140805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:37:49.858263+0000","scrub_duration":0.035369036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1021,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.860338+0000","last_change":"2023-12-08T00:18:21.860992+0000","last_active":"2023-12-08T00:18:21.860338+0000","last_peered":"2023-12-08T00:18:21.860338+0000","last_clean":"2023-12-08T00:18:21.860338+0000","last_became_active":"2023-12-07T23:47:18.946564+0000","last_became_peered":"2023-12-07T23:47:18.946564+0000","last_unstale":"2023-12-08T00:18:21.860338+0000","last_undegraded":"2023-12-08T00:18:21.860338+0000","last_fullsized":"2023-12-08T00:18:21.860338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.858949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.858949+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.858949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:22:09.518786+0000","scrub_duration":0.075188711000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":2,"acting":2,"num_store_stats":2},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477125,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81244,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290596,"statfs":{"total":96636764160,"available":96553570304,"internally_reserved":0,"allocated":5787648,"data_stored":2681864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":12.305999999999999,"15min":16.677},"min":{"1min":5.2380000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"max":{"1min":255.59,"5min":255.59,"15min":272.77100000000002},"last":6.0049999999999999},{"interface":"front","average":{"1min":26.963999999999999,"5min":14.023,"15min":19.472000000000001},"min":{"1min":6.8899999999999997,"5min":4.9900000000000002,"15min":4.9649999999999999},"max":{"1min":224.32400000000001,"5min":224.32400000000001,"15min":288.42000000000002},"last":10.52}]},{"osd":1,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":21.672000000000001,"5min":14.118,"15min":21.623999999999999},"min":{"1min":8.4749999999999996,"5min":6.2279999999999998,"15min":5.7649999999999997},"max":{"1min":122.941,"5min":122.941,"15min":320.57900000000001},"last":7.0679999999999996},{"interface":"front","average":{"1min":18.599,"5min":14.475,"15min":23.824000000000002},"min":{"1min":7.9180000000000001,"5min":6.7279999999999998,"15min":5.5670000000000002},"max":{"1min":60.479999999999997,"5min":91.882000000000005,"15min":461.51799999999997},"last":11.198}]},{"osd":2,"last update":"Fri Dec 8 00:17:38 2023","interfaces":[{"interface":"back","average":{"1min":11.853,"5min":9.7810000000000006,"15min":15.744},"min":{"1min":7.343,"5min":6.0389999999999997,"15min":5.6959999999999997},"max":{"1min":66.941999999999993,"5min":66.941999999999993,"15min":246.47499999999999},"last":8.2479999999999993},{"interface":"front","average":{"1min":17.085000000000001,"5min":14.343,"15min":22.997},"min":{"1min":8.7669999999999995,"5min":8.7279999999999998,"15min":7.3079999999999998},"max":{"1min":69.078999999999994,"5min":69.078999999999994,"15min":356.02100000000002},"last":14.663}]},{"osd":4,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":27.315000000000001,"5min":17.056000000000001,"15min":20.420999999999999},"min":{"1min":11.606,"5min":7.048,"15min":6.7690000000000001},"max":{"1min":160.06100000000001,"5min":160.06100000000001,"15min":180.76400000000001},"last":11.502000000000001},{"interface":"front","average":{"1min":29.664000000000001,"5min":17.638999999999999,"15min":23.712},"min":{"1min":11.324,"5min":8.3040000000000003,"15min":6.2990000000000004},"max":{"1min":173.029,"5min":173.029,"15min":223.91300000000001},"last":12.109}]},{"osd":5,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":34.090000000000003,"5min":18.847000000000001,"15min":25.256},"min":{"1min":12.975,"5min":8.9920000000000009,"15min":7.2889999999999997},"max":{"1min":219.87899999999999,"5min":219.87899999999999,"15min":251.892},"last":12.843999999999999},{"interface":"front","average":{"1min":39.415999999999997,"5min":24.015000000000001,"15min":29.675999999999998},"min":{"1min":14.428000000000001,"5min":11.711,"15min":10.568},"max":{"1min":259.93099999999998,"5min":265.69799999999998,"15min":279.791},"last":16.427}]},{"osd":6,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":26.574000000000002,"5min":17.271000000000001,"15min":20.309000000000001},"min":{"1min":8.8170000000000002,"5min":7.2850000000000001,"15min":7.2850000000000001},"max":{"1min":162.42699999999999,"5min":214.774,"15min":214.774},"last":14.010999999999999},{"interface":"front","average":{"1min":31.245000000000001,"5min":21.231999999999999,"15min":25.077999999999999},"min":{"1min":11.41,"5min":10.428000000000001,"15min":8.1440000000000001},"max":{"1min":169.095,"5min":229.54300000000001,"15min":229.54300000000001},"last":15.702}]},{"osd":7,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":38.497,"5min":21.491,"15min":29.875},"min":{"1min":14.987,"5min":8.923,"15min":8.923},"max":{"1min":246.601,"5min":246.601,"15min":281.31},"last":18.462},{"interface":"front","average":{"1min":30.427,"5min":17.623000000000001,"15min":23.940000000000001},"min":{"1min":12.416,"5min":8.8010000000000002,"15min":7.649},"max":{"1min":163.34700000000001,"5min":163.34700000000001,"15min":255.72499999999999},"last":17.18}]}]},{"osd":2,"up_from":16,"seq":68719477126,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68528,"kb_used_data":5416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303312,"statfs":{"total":96636764160,"available":96566591488,"internally_reserved":0,"allocated":5545984,"data_stored":2548512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":11.218,"5min":9.1470000000000002,"15min":14.103},"min":{"1min":5.5700000000000003,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":41.286000000000001,"5min":41.286000000000001,"15min":249.42699999999999},"last":4.8259999999999996},{"interface":"front","average":{"1min":12.199999999999999,"5min":10.103999999999999,"15min":15.738},"min":{"1min":5.5019999999999998,"5min":5.5019999999999998,"15min":5.1310000000000002},"max":{"1min":42.930999999999997,"5min":42.930999999999997,"15min":241.45599999999999},"last":10.223000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":17.216999999999999,"5min":13.097,"15min":18.321000000000002},"min":{"1min":6.6020000000000003,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":87.658000000000001,"5min":87.658000000000001,"15min":234.279},"last":15.930999999999999},{"interface":"front","average":{"1min":14.349,"5min":11.438000000000001,"15min":16.529},"min":{"1min":6.0640000000000001,"5min":6.0640000000000001,"15min":5.9109999999999996},"max":{"1min":82.905000000000001,"5min":82.905000000000001,"15min":258.61599999999999},"last":11.74}]},{"osd":3,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":13.989000000000001,"5min":11.375999999999999,"15min":18.414999999999999},"min":{"1min":6.8099999999999996,"5min":6.2770000000000001,"15min":6.1550000000000002},"max":{"1min":31.184000000000001,"5min":31.184000000000001,"15min":287.56299999999999},"last":8.3059999999999992},{"interface":"front","average":{"1min":15.461,"5min":14.363,"15min":20.439},"min":{"1min":8.6760000000000002,"5min":8.3059999999999992,"15min":7.149},"max":{"1min":36.813000000000002,"5min":36.813000000000002,"15min":223.20400000000001},"last":17.548999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":18.015999999999998,"5min":15.794,"15min":21.940000000000001},"min":{"1min":11.374000000000001,"5min":8.9220000000000006,"15min":8.1539999999999999},"max":{"1min":40.554000000000002,"5min":40.554000000000002,"15min":213.88},"last":16.701000000000001},{"interface":"front","average":{"1min":13.412000000000001,"5min":12.475,"15min":18.739999999999998},"min":{"1min":8.3490000000000002,"5min":6.7450000000000001,"15min":6.2649999999999997},"max":{"1min":35.106999999999999,"5min":35.106999999999999,"15min":212.494},"last":10.548}]},{"osd":5,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":21.234000000000002,"5min":17.533999999999999,"15min":22.791},"min":{"1min":12.539999999999999,"5min":9.0410000000000004,"15min":8.5470000000000006},"max":{"1min":88.070999999999998,"5min":88.070999999999998,"15min":204.60599999999999},"last":18.094999999999999},{"interface":"front","average":{"1min":17.984000000000002,"5min":14.983000000000001,"15min":19.736999999999998},"min":{"1min":9.3889999999999993,"5min":8.7360000000000007,"15min":7.0039999999999996},"max":{"1min":96.875,"5min":96.875,"15min":248.63499999999999},"last":12.361000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":16.193999999999999,"15min":19.606999999999999},"min":{"1min":10.353999999999999,"5min":10.353999999999999,"15min":7.4240000000000004},"max":{"1min":41.552999999999997,"5min":41.552999999999997,"15min":232.48099999999999},"last":13.06},{"interface":"front","average":{"1min":17.151,"5min":13.273,"15min":19.757000000000001},"min":{"1min":8.2100000000000009,"5min":7.423,"15min":7.423},"max":{"1min":35.909999999999997,"5min":35.909999999999997,"15min":231.422},"last":9.9700000000000006}]},{"osd":7,"last update":"Fri Dec 8 00:18:13 2023","interfaces":[{"interface":"back","average":{"1min":17.989000000000001,"5min":14.842000000000001,"15min":23.449999999999999},"min":{"1min":12.250999999999999,"5min":8.6010000000000009,"15min":8.6010000000000009},"max":{"1min":33.475999999999999,"5min":33.475999999999999,"15min":350.52199999999999},"last":11.193},{"interface":"front","average":{"1min":17.739000000000001,"5min":17.244,"15min":24.251000000000001},"min":{"1min":11.827,"5min":11.411,"15min":10.475},"max":{"1min":38.564999999999998,"5min":38.564999999999998,"15min":235.19200000000001},"last":14.956}]}]},{"osd":0,"up_from":16,"seq":68719477125,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68764,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303076,"statfs":{"total":96636764160,"available":96566349824,"internally_reserved":0,"allocated":5787648,"data_stored":2681864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 8 00:17:29 2023","interfaces":[{"interface":"back","average":{"1min":10.356,"5min":9.7119999999999997,"15min":18.596},"min":{"1min":5.2789999999999999,"5min":4.827,"15min":4.5830000000000002},"max":{"1min":38.834000000000003,"5min":38.834000000000003,"15min":317.99599999999998},"last":11.393000000000001},{"interface":"front","average":{"1min":11.698,"5min":9.2210000000000001,"15min":17.318999999999999},"min":{"1min":6.6840000000000002,"5min":4.9139999999999997,"15min":4.9139999999999997},"max":{"1min":49.262,"5min":49.262,"15min":313.14400000000001},"last":18.018999999999998}]},{"osd":2,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":18.568000000000001,"5min":17.286000000000001,"15min":20.753},"min":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":118.601,"5min":196.96700000000001,"15min":228.02500000000001},"last":7.8490000000000002},{"interface":"front","average":{"1min":17.353999999999999,"5min":15.314,"15min":18.536999999999999},"min":{"1min":7.6559999999999997,"5min":5.7530000000000001,"15min":5.6059999999999999},"max":{"1min":43.734000000000002,"5min":168.91300000000001,"15min":197.78899999999999},"last":17.116}]},{"osd":3,"last update":"Fri Dec 8 00:18:11 2023","interfaces":[{"interface":"back","average":{"1min":18.135999999999999,"5min":14.326000000000001,"15min":22.596},"min":{"1min":8.6419999999999995,"5min":8.0510000000000002,"15min":7.0069999999999997},"max":{"1min":115.229,"5min":115.229,"15min":239.50999999999999},"last":24.498000000000001},{"interface":"front","average":{"1min":17.655999999999999,"5min":16.533999999999999,"15min":23.977},"min":{"1min":12.94,"5min":10.667,"15min":9.141},"max":{"1min":34.673000000000002,"5min":34.673000000000002,"15min":241.839},"last":22.582000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:18:11 2023","interfaces":[{"interface":"back","average":{"1min":12.961,"5min":9.7360000000000007,"15min":14.757},"min":{"1min":6.7119999999999997,"5min":5.5570000000000004,"15min":5.5430000000000001},"max":{"1min":41.896999999999998,"5min":41.896999999999998,"15min":211.452},"last":10.664},{"interface":"front","average":{"1min":18.591000000000001,"5min":17.117999999999999,"15min":22.282},"min":{"1min":11.193,"5min":10.212999999999999,"15min":9.3030000000000008},"max":{"1min":37.057000000000002,"5min":37.057000000000002,"15min":212.459},"last":11.955}]},{"osd":5,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":17.085000000000001,"5min":15.750999999999999,"15min":19.074000000000002},"min":{"1min":9.8079999999999998,"5min":7.8970000000000002,"15min":6.2560000000000002},"max":{"1min":33.323999999999998,"5min":175.68100000000001,"15min":175.68100000000001},"last":19.222000000000001},{"interface":"front","average":{"1min":12.287000000000001,"5min":13.906000000000001,"15min":19.5},"min":{"1min":8.6560000000000006,"5min":6.7789999999999999,"15min":6.6660000000000004},"max":{"1min":31.010000000000002,"5min":186.625,"15min":186.625},"last":14.957000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":17.896999999999998,"15min":25.821000000000002},"min":{"1min":11.704000000000001,"5min":11.406000000000001,"15min":11.166},"max":{"1min":28.852,"5min":30.727,"15min":227.892},"last":12.657},{"interface":"front","average":{"1min":16.443999999999999,"5min":14.975,"15min":20.138000000000002},"min":{"1min":9.5009999999999994,"5min":8.2919999999999998,"15min":8.2919999999999998},"max":{"1min":32.412999999999997,"5min":44.432000000000002,"15min":153.59700000000001},"last":20.501000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":20.614000000000001,"5min":17.145,"15min":24.812000000000001},"min":{"1min":13.722,"5min":9.3550000000000004,"15min":8.9369999999999994},"max":{"1min":49.183,"5min":49.183,"15min":264.44600000000003},"last":21.559000000000001},{"interface":"front","average":{"1min":13.4,"5min":13.119,"15min":20.073},"min":{"1min":8.6920000000000002,"5min":7.5350000000000001,"15min":7.5},"max":{"1min":27.434000000000001,"5min":41.091000000000001,"15min":243.84899999999999},"last":16.545000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542531,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68924,"kb_used_data":5812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302916,"statfs":{"total":96636764160,"available":96566185984,"internally_reserved":0,"allocated":5951488,"data_stored":2946357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:16 2023","interfaces":[{"interface":"back","average":{"1min":8.8390000000000004,"5min":9.2059999999999995,"15min":16.134},"min":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.2539999999999996},"max":{"1min":14.042999999999999,"5min":18.986999999999998,"15min":324.46600000000001},"last":8.0999999999999996},{"interface":"front","average":{"1min":12.923999999999999,"5min":13.388999999999999,"15min":21.181999999999999},"min":{"1min":7.9139999999999997,"5min":6.4950000000000001,"15min":5.5369999999999999},"max":{"1min":31.443000000000001,"5min":31.443000000000001,"15min":342.39299999999997},"last":8.8160000000000007}]},{"osd":2,"last update":"Fri Dec 8 00:18:16 2023","interfaces":[{"interface":"back","average":{"1min":18.105,"5min":11.507,"15min":19.349},"min":{"1min":6.3029999999999999,"5min":4.7409999999999997,"15min":4.7409999999999997},"max":{"1min":109.697,"5min":109.697,"15min":288.863},"last":7.3280000000000003},{"interface":"front","average":{"1min":32.765999999999998,"5min":17.178999999999998,"15min":24.306000000000001},"min":{"1min":10.532999999999999,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":330.75900000000001,"5min":330.75900000000001,"15min":343.49900000000002},"last":15.282999999999999}]},{"osd":3,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":13.263,"5min":14.33,"15min":22.765000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"max":{"1min":17.984999999999999,"5min":20.395,"15min":345.07600000000002},"last":11.491},{"interface":"front","average":{"1min":10.297000000000001,"5min":9.077,"15min":15.353},"min":{"1min":7.5209999999999999,"5min":6.1050000000000004,"15min":5.3090000000000002},"max":{"1min":15.089,"5min":16.981999999999999,"15min":174.20599999999999},"last":6.5670000000000002}]},{"osd":4,"last update":"Fri Dec 8 00:18:16 2023","interfaces":[{"interface":"back","average":{"1min":13.456,"5min":11.994,"15min":18.782},"min":{"1min":6.8499999999999996,"5min":6.1970000000000001,"15min":6.0419999999999998},"max":{"1min":32.310000000000002,"5min":32.310000000000002,"15min":290.358},"last":12.664},{"interface":"front","average":{"1min":12.930999999999999,"5min":12.369,"15min":18.876000000000001},"min":{"1min":8.3000000000000007,"5min":6.1909999999999998,"15min":6.0469999999999997},"max":{"1min":20.007000000000001,"5min":20.007000000000001,"15min":169.90799999999999},"last":14.699999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":32.521999999999998,"5min":19.515000000000001,"15min":24.640999999999998},"min":{"1min":10.952999999999999,"5min":10.938000000000001,"15min":7.7690000000000001},"max":{"1min":333.11799999999999,"5min":333.11799999999999,"15min":371.50599999999997},"last":13.288},{"interface":"front","average":{"1min":28.724,"5min":15.845000000000001,"15min":19.704000000000001},"min":{"1min":9.0389999999999997,"5min":7.6150000000000002,"15min":7.282},"max":{"1min":298.12099999999998,"5min":298.12099999999998,"15min":298.12099999999998},"last":10.85}]},{"osd":6,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":14.052,"5min":13.733000000000001,"15min":19.829000000000001},"min":{"1min":9.0879999999999992,"5min":7.3490000000000002,"15min":7.3490000000000002},"max":{"1min":19.085000000000001,"5min":24.93,"15min":292.72300000000001},"last":9.4969999999999999},{"interface":"front","average":{"1min":15.968999999999999,"5min":14.984,"15min":22.007999999999999},"min":{"1min":10.695,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":18.954999999999998,"5min":26.942,"15min":173.17599999999999},"last":12.082000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:18:04 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":16.228999999999999,"15min":26.417000000000002},"min":{"1min":10.32,"5min":9.3789999999999996,"15min":9.3789999999999996},"max":{"1min":94.143000000000001,"5min":94.143000000000001,"15min":350.50999999999999},"last":17.411000000000001},{"interface":"front","average":{"1min":24.823,"5min":18.376999999999999,"15min":24.126000000000001},"min":{"1min":12.754,"5min":12.005000000000001,"15min":8.3659999999999997},"max":{"1min":90.555999999999997,"5min":90.555999999999997,"15min":304.64499999999998},"last":16.388999999999999}]}]},{"osd":7,"up_from":13,"seq":55834575238,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68528,"kb_used_data":5416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303312,"statfs":{"total":96636764160,"available":96566591488,"internally_reserved":0,"allocated":5545984,"data_stored":2548512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":10.093999999999999,"15min":19.027000000000001},"min":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":4.5179999999999998},"max":{"1min":73.165999999999997,"5min":73.165999999999997,"15min":259.01499999999999},"last":8.5530000000000008},{"interface":"front","average":{"1min":16.966000000000001,"5min":14.547000000000001,"15min":23.523},"min":{"1min":9.3879999999999999,"5min":6.1369999999999996,"15min":6.016},"max":{"1min":73.781000000000006,"5min":73.781000000000006,"15min":292.73899999999998},"last":16.635000000000002}]},{"osd":1,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":11.380000000000001,"5min":10.147,"15min":19.216000000000001},"min":{"1min":4.3289999999999997,"5min":4.3289999999999997,"15min":4.3289999999999997},"max":{"1min":28.245000000000001,"5min":28.245000000000001,"15min":414.57100000000003},"last":6.2880000000000003},{"interface":"front","average":{"1min":14.326000000000001,"5min":11.039,"15min":17.794},"min":{"1min":6.327,"5min":6.327,"15min":5.984},"max":{"1min":38.805,"5min":38.805,"15min":333.53800000000001},"last":9.9499999999999993}]},{"osd":2,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":18.044,"5min":13.077,"15min":21.619},"min":{"1min":7.3449999999999998,"5min":6.75,"15min":6.75},"max":{"1min":94.254000000000005,"5min":94.254000000000005,"15min":286.63499999999999},"last":9.3930000000000007},{"interface":"front","average":{"1min":22.062000000000001,"5min":14.420999999999999,"15min":21.707999999999998},"min":{"1min":11.499000000000001,"5min":7.0540000000000003,"15min":6.1280000000000001},"max":{"1min":96.084000000000003,"5min":96.084000000000003,"15min":299.54300000000001},"last":10.664}]},{"osd":3,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":15.101000000000001,"5min":13.878,"15min":23.503},"min":{"1min":7.819,"5min":7.0810000000000004,"15min":6.6289999999999996},"max":{"1min":37.604999999999997,"5min":37.604999999999997,"15min":391.23200000000003},"last":13.257999999999999},{"interface":"front","average":{"1min":17.687000000000001,"5min":16.838000000000001,"15min":25.504000000000001},"min":{"1min":11.420999999999999,"5min":10.843,"15min":8.3949999999999996},"max":{"1min":27.238,"5min":27.238,"15min":362.20299999999997},"last":17.411000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":17.986000000000001,"5min":17.036999999999999,"15min":23.123999999999999},"min":{"1min":9.2989999999999995,"5min":7.6710000000000003,"15min":7.6710000000000003},"max":{"1min":31.456,"5min":31.456,"15min":293.358},"last":15.771000000000001},{"interface":"front","average":{"1min":17.670000000000002,"5min":14.42,"15min":20.253},"min":{"1min":7.6959999999999997,"5min":7.6959999999999997,"15min":6.6769999999999996},"max":{"1min":38.436,"5min":38.436,"15min":275.31799999999998},"last":12.621}]},{"osd":5,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":22.408000000000001,"5min":18.739999999999998,"15min":26.800999999999998},"min":{"1min":13.476000000000001,"5min":9.6720000000000006,"15min":7.7380000000000004},"max":{"1min":97.998000000000005,"5min":97.998000000000005,"15min":294.56400000000002},"last":14.992000000000001},{"interface":"front","average":{"1min":17.391999999999999,"5min":12.614000000000001,"15min":23.106000000000002},"min":{"1min":7.1719999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"max":{"1min":143.78100000000001,"5min":143.78100000000001,"15min":280.83600000000001},"last":7.6870000000000003}]},{"osd":6,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":18.853000000000002,"5min":16.222999999999999,"15min":24.637},"min":{"1min":13.026999999999999,"5min":10.958,"15min":8.5579999999999998},"max":{"1min":40.783999999999999,"5min":40.783999999999999,"15min":289.798},"last":11.284000000000001},{"interface":"front","average":{"1min":15.396000000000001,"5min":14.058,"15min":23.053000000000001},"min":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"max":{"1min":40.825000000000003,"5min":40.825000000000003,"15min":303.30900000000003},"last":11.904999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575237,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85408,"kb_used_data":5656,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94286432,"statfs":{"total":96636764160,"available":96549306368,"internally_reserved":0,"allocated":5791744,"data_stored":2681869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:20 2023","interfaces":[{"interface":"back","average":{"1min":10.545,"5min":10.394,"15min":14.253},"min":{"1min":4.2409999999999997,"5min":4.2409999999999997,"15min":4.1920000000000002},"max":{"1min":23.716999999999999,"5min":23.716999999999999,"15min":168.34100000000001},"last":11.050000000000001},{"interface":"front","average":{"1min":8.9920000000000009,"5min":8.8610000000000007,"15min":13.792},"min":{"1min":4.3869999999999996,"5min":4.3869999999999996,"15min":4.3869999999999996},"max":{"1min":27.768000000000001,"5min":27.768000000000001,"15min":172.452},"last":5.3230000000000004}]},{"osd":1,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":12.702999999999999,"5min":13.539,"15min":16.152000000000001},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.1870000000000003},"max":{"1min":18.241,"5min":20.835000000000001,"15min":206.43899999999999},"last":13.929},{"interface":"front","average":{"1min":16.012,"5min":16.016999999999999,"15min":19.134},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":7.2080000000000002},"max":{"1min":35.750999999999998,"5min":35.750999999999998,"15min":155.94},"last":14.606999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:18:07 2023","interfaces":[{"interface":"back","average":{"1min":10.789,"5min":11.645,"15min":15.612},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":5.8940000000000001},"max":{"1min":21.337,"5min":21.337,"15min":168.23599999999999},"last":12.786},{"interface":"front","average":{"1min":9.5329999999999995,"5min":9.9109999999999996,"15min":15.417999999999999},"min":{"1min":6.3979999999999997,"5min":4.8540000000000001,"15min":4.8540000000000001},"max":{"1min":20.876999999999999,"5min":20.876999999999999,"15min":352.52800000000002},"last":7.6360000000000001}]},{"osd":3,"last update":"Fri Dec 8 00:17:39 2023","interfaces":[{"interface":"back","average":{"1min":11.417999999999999,"5min":11.669,"15min":17.757999999999999},"min":{"1min":6.867,"5min":6.867,"15min":5.8789999999999996},"max":{"1min":21.459,"5min":21.459,"15min":286.92399999999998},"last":12.247999999999999},{"interface":"front","average":{"1min":17.001000000000001,"5min":15.720000000000001,"15min":22.109000000000002},"min":{"1min":9.8889999999999993,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":36.771000000000001,"5min":36.771000000000001,"15min":265.44400000000002},"last":16.004000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:17:46 2023","interfaces":[{"interface":"back","average":{"1min":13.489000000000001,"5min":13.015000000000001,"15min":18.609000000000002},"min":{"1min":7.3319999999999999,"5min":6.7389999999999999,"15min":6.7389999999999999},"max":{"1min":33.700000000000003,"5min":33.700000000000003,"15min":353.452},"last":9.9179999999999993},{"interface":"front","average":{"1min":17.852,"5min":16.946000000000002,"15min":23.024000000000001},"min":{"1min":9.7469999999999999,"5min":9.7469999999999999,"15min":8.8499999999999996},"max":{"1min":37.723999999999997,"5min":37.723999999999997,"15min":341.87299999999999},"last":17.338000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:18:07 2023","interfaces":[{"interface":"back","average":{"1min":20.192,"5min":18.701000000000001,"15min":23.984000000000002},"min":{"1min":8.9990000000000006,"5min":8.9990000000000006,"15min":8.9990000000000006},"max":{"1min":70.762,"5min":70.762,"15min":158.316},"last":17.952000000000002},{"interface":"front","average":{"1min":15.691000000000001,"5min":12.763,"15min":17.699999999999999},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.056},"max":{"1min":66.159000000000006,"5min":66.159000000000006,"15min":157.48500000000001},"last":10.196}]},{"osd":7,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":13.949999999999999,"5min":14.15,"15min":18.186},"min":{"1min":8.5719999999999992,"5min":8.5719999999999992,"15min":8.4930000000000003},"max":{"1min":30.861000000000001,"5min":30.861000000000001,"15min":152.49600000000001},"last":15.279},{"interface":"front","average":{"1min":18.798999999999999,"5min":18.960000000000001,"15min":23.832999999999998},"min":{"1min":11.927,"5min":11.927,"15min":11.388},"max":{"1min":38.762,"5min":38.762,"15min":299.93099999999998},"last":18.919}]}]},{"osd":5,"up_from":13,"seq":55834575236,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73316,"kb_used_data":6044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298524,"statfs":{"total":96636764160,"available":96561688576,"internally_reserved":0,"allocated":6189056,"data_stored":3079704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":8.5069999999999997,"15min":14.414999999999999},"min":{"1min":4.4740000000000002,"5min":4.226,"15min":4.226},"max":{"1min":55.738,"5min":55.738,"15min":401.80099999999999},"last":7.0949999999999998},{"interface":"front","average":{"1min":18.251000000000001,"5min":11.83,"15min":15.614000000000001},"min":{"1min":6.1360000000000001,"5min":4.9820000000000002,"15min":4.7690000000000001},"max":{"1min":83.034000000000006,"5min":83.034000000000006,"15min":139.387},"last":6.4269999999999996}]},{"osd":1,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":16.733000000000001,"5min":13.003,"15min":17.408000000000001},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":5.5540000000000003},"max":{"1min":85.837999999999994,"5min":85.837999999999994,"15min":168.71299999999999},"last":7.8300000000000001},{"interface":"front","average":{"1min":24.350000000000001,"5min":17.399999999999999,"15min":23.219999999999999},"min":{"1min":11.083,"5min":10.079000000000001,"15min":10.079000000000001},"max":{"1min":87.222999999999999,"5min":87.222999999999999,"15min":297.87299999999999},"last":11.625999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:17:34 2023","interfaces":[{"interface":"back","average":{"1min":10.785,"5min":11.074999999999999,"15min":16.484999999999999},"min":{"1min":5.5460000000000003,"5min":5.5460000000000003,"15min":4.6980000000000004},"max":{"1min":17.513000000000002,"5min":20.484999999999999,"15min":222.297},"last":13.709},{"interface":"front","average":{"1min":16.869,"5min":12.349,"15min":17.806999999999999},"min":{"1min":7.9729999999999999,"5min":6.0209999999999999,"15min":5.7789999999999999},"max":{"1min":88.489000000000004,"5min":88.489000000000004,"15min":402.50200000000001},"last":10.282}]},{"osd":3,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.141999999999999,"5min":14.019,"15min":20.75},"min":{"1min":8,"5min":6.6630000000000003,"15min":6.5190000000000001},"max":{"1min":86.459000000000003,"5min":86.459000000000003,"15min":295.20499999999998},"last":21.425000000000001},{"interface":"front","average":{"1min":23.57,"5min":16.802,"15min":23.039999999999999},"min":{"1min":7.7939999999999996,"5min":7.7939999999999996,"15min":7.7939999999999996},"max":{"1min":95.340000000000003,"5min":95.340000000000003,"15min":154.19200000000001},"last":22.986999999999998}]},{"osd":4,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":24.349,"5min":18.375,"15min":23.802},"min":{"1min":9.2729999999999997,"5min":9.2729999999999997,"15min":9.0760000000000005},"max":{"1min":84.105999999999995,"5min":84.105999999999995,"15min":337.97199999999998},"last":10.999000000000001},{"interface":"front","average":{"1min":20.994,"5min":15.461,"15min":19.864000000000001},"min":{"1min":11.005000000000001,"5min":9.0180000000000007,"15min":6.7350000000000003},"max":{"1min":67.284000000000006,"5min":67.284000000000006,"15min":169.846},"last":14.545999999999999}]},{"osd":6,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.216999999999999,"5min":17.192,"15min":19.756},"min":{"1min":9.3260000000000005,"5min":8.4109999999999996,"15min":7.4109999999999996},"max":{"1min":67.876999999999995,"5min":67.876999999999995,"15min":169.93199999999999},"last":15.356},{"interface":"front","average":{"1min":26.831,"5min":19.538,"15min":24.748999999999999},"min":{"1min":14.941000000000001,"5min":10.512,"15min":8.7810000000000006},"max":{"1min":89.180000000000007,"5min":89.180000000000007,"15min":395.5},"last":12.268000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":25.442,"5min":14.970000000000001,"15min":23.666},"min":{"1min":13.343,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":81.359999999999999,"5min":81.359999999999999,"15min":417.26299999999998},"last":16.177},{"interface":"front","average":{"1min":20.204000000000001,"5min":17.381,"15min":21.201000000000001},"min":{"1min":7.5359999999999996,"5min":7.5359999999999996,"15min":6.5780000000000003},"max":{"1min":84.742000000000004,"5min":84.742000000000004,"15min":352.34100000000001},"last":9.2089999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575236,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68764,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303076,"statfs":{"total":96636764160,"available":96566349824,"internally_reserved":0,"allocated":5787648,"data_stored":2681864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.9109999999999996,"15min":15.105},"min":{"1min":4.9039999999999999,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":19.489999999999998,"5min":19.489999999999998,"15min":218.566},"last":6.4630000000000001},{"interface":"front","average":{"1min":9.6110000000000007,"5min":10.023,"15min":16.123999999999999},"min":{"1min":5.4039999999999999,"5min":5.1520000000000001,"15min":4.8300000000000001},"max":{"1min":19.678999999999998,"5min":19.678999999999998,"15min":299.358},"last":12.98}]},{"osd":1,"last update":"Fri Dec 8 00:18:18 2023","interfaces":[{"interface":"back","average":{"1min":14.259,"5min":12.289,"15min":16.882000000000001},"min":{"1min":5.8319999999999999,"5min":5.8319999999999999,"15min":5.8319999999999999},"max":{"1min":58.145000000000003,"5min":58.145000000000003,"15min":291.13200000000001},"last":14.656000000000001},{"interface":"front","average":{"1min":13.249000000000001,"5min":11.358000000000001,"15min":14.342000000000001},"min":{"1min":7.0549999999999997,"5min":5.3490000000000002,"15min":4.9509999999999996},"max":{"1min":55.795000000000002,"5min":55.795000000000002,"15min":119.005},"last":7.5149999999999997}]},{"osd":2,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":11.776999999999999,"15min":17.225000000000001},"min":{"1min":7.2880000000000003,"5min":6.8819999999999997,"15min":6.1539999999999999},"max":{"1min":86.319000000000003,"5min":86.319000000000003,"15min":300.22399999999999},"last":8.1669999999999998},{"interface":"front","average":{"1min":15.244,"5min":14.737,"15min":20.122},"min":{"1min":11.385,"5min":8.7119999999999997,"15min":6.7009999999999996},"max":{"1min":26.940999999999999,"5min":26.940999999999999,"15min":258.58499999999998},"last":10.35}]},{"osd":3,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.311,"5min":14.824999999999999,"15min":22.536999999999999},"min":{"1min":9.6920000000000002,"5min":7.4279999999999999,"15min":6.4219999999999997},"max":{"1min":51.329999999999998,"5min":51.329999999999998,"15min":400.41699999999997},"last":17.777000000000001},{"interface":"front","average":{"1min":15.747999999999999,"5min":13.279,"15min":19.422000000000001},"min":{"1min":9.8249999999999993,"5min":7.0460000000000003,"15min":7.0460000000000003},"max":{"1min":49.639000000000003,"5min":49.639000000000003,"15min":273.839},"last":15.353999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:30 2023","interfaces":[{"interface":"back","average":{"1min":16.038,"5min":15.727,"15min":22.605},"min":{"1min":10.035,"5min":7.4939999999999998,"15min":6.7569999999999997},"max":{"1min":33.755000000000003,"5min":33.755000000000003,"15min":397.96199999999999},"last":16.349},{"interface":"front","average":{"1min":17.222999999999999,"5min":14.757,"15min":20.649000000000001},"min":{"1min":10.176,"5min":7.0129999999999999,"15min":6.9900000000000002},"max":{"1min":59.944000000000003,"5min":59.944000000000003,"15min":291.952},"last":14.878}]},{"osd":6,"last update":"Fri Dec 8 00:17:47 2023","interfaces":[{"interface":"back","average":{"1min":19.376999999999999,"5min":15.308999999999999,"15min":20.122},"min":{"1min":8.5579999999999998,"5min":7.8940000000000001,"15min":6.1399999999999997},"max":{"1min":50.722999999999999,"5min":50.722999999999999,"15min":296.37},"last":12.177},{"interface":"front","average":{"1min":25.309000000000001,"5min":17.347000000000001,"15min":21.280000000000001},"min":{"1min":11.715,"5min":8.9130000000000003,"15min":7.8860000000000001},"max":{"1min":111.55,"5min":111.55,"15min":232.64099999999999},"last":11.365}]},{"osd":7,"last update":"Fri Dec 8 00:18:18 2023","interfaces":[{"interface":"back","average":{"1min":17.148,"5min":17.062999999999999,"15min":22.404},"min":{"1min":10.853,"5min":9.6539999999999999,"15min":7.8659999999999997},"max":{"1min":25.847999999999999,"5min":25.847999999999999,"15min":235.828},"last":16.088000000000001},{"interface":"front","average":{"1min":16.129999999999999,"5min":16.532,"15min":22.901},"min":{"1min":10.551,"5min":9.6530000000000005,"15min":9.0869999999999997},"max":{"1min":27.341000000000001,"5min":27.341000000000001,"15min":298.74599999999998},"last":20.513000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T00:18:34.067 INFO:tasks.ceph:pgid 15.3 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:34.068 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:34.068 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:34.068 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2023-12-07T23:47:10.890608+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=23, tm_min=47, tm_sec=10, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=8, tm_hour=0, tm_min=18, tm_sec=5, tm_wday=4, tm_yday=342, tm_isdst=0) 2023-12-08T00:18:34.068 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-08T00:18:36.073 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:32.547 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.077 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:32.550 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.097 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:32.570 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.123 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:32.569 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.126 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:32.572 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.130 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:32.603 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.135 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:32.608 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.142 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:32.588 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:36.149 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:32.623 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.087 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:37.561 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.089 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:37.563 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.101 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:37.575 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.139 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:37.585 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.156 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:37.601 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.163 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:37.636 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.165 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:37.638 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.173 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:37.619 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:41.176 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:37.649 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:54.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.186+0000 7fc345d47640 10 monclient: get_monmap_and_config 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.186+0000 7fc345d47640 10 monclient: build_initial_monmap 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.187+0000 7fc345d47640 10 monclient: monmap: 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:54.187143+0000 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:54.187143+0000 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.187+0000 7fc345d47640 1 Processor -- start 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.187+0000 7fc345d47640 1 -- start start 2023-12-08T00:18:54.190 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.187+0000 7fc345d47640 10 monclient: init 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc34006bf50 0x7fc34013f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient(hunting): picked mon.noname-c con 0x7fc34006bf50 addr v2:172.21.15.184:3300/0 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc34013f590 0x7fc3401453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient(hunting): picked mon.noname-a con 0x7fc34013f590 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc340145930 addr v1:172.21.15.184:6789/0 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3400ad4b0 con 0x7fc340145930 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34006c330 con 0x7fc34013f590 2023-12-08T00:18:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3400ae280 con 0x7fc34006bf50 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc345d47640 10 monclient(hunting): authenticate will time out at 3334.375437s 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.188+0000 7fc33effd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc34013f590 0x7fc3401453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc34013f590 0x7fc3401453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50128/0 (socket says 172.21.15.129:50128) 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 1 -- 172.21.15.129:0/2939742075 learned_addr learned my addr 172.21.15.129:0/2939742075 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 10 monclient(hunting): get_auth_request con 0x7fc34013f590 auth_method 0 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33f7fe640 1 --2- 172.21.15.129:0/2939742075 >> v2:172.21.15.184:3300/0 conn(0x7fc34006bf50 0x7fc34013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.194 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.189+0000 7fc33effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.190+0000 7fc33f7fe640 10 monclient(hunting): get_auth_request con 0x7fc34006bf50 auth_method 0 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.190+0000 7fc33f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.190+0000 7fc33f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.190+0000 7fc33f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 10 monclient(hunting): handle_auth_done global_id 15816 payload 274 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 1 -- 172.21.15.129:0/2939742075 >> v2:172.21.15.184:3300/0 conn(0x7fc34006bf50 msgr2=0x7fc34013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 1 --2- 172.21.15.129:0/2939742075 >> v2:172.21.15.184:3300/0 conn(0x7fc34006bf50 0x7fc34013f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 1 -- 172.21.15.129:0/2939742075 >> v1:172.21.15.184:6789/0 conn(0x7fc340145930 legacy=0x7fc3400b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 10 monclient: _finish_hunting 0 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 1 monclient: found mon.noname-a 2023-12-08T00:18:54.195 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:54.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.191+0000 7fc33effd640 1 -- 172.21.15.129:0/2939742075 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3400ae4f0 con 0x7fc34013f590 2023-12-08T00:18:54.196 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.193+0000 7fc33effd640 1 --2- 172.21.15.129:0/2939742075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc34013f590 0x7fc3401453f0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7fc34006c610 tx=0x7fc33402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcca99813f2072c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc33e7fc640 1 -- 172.21.15.129:0/2939742075 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334031870 con 0x7fc34013f590 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc33e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc33e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc33e7fc640 10 monclient: dump: 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.197 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc33e7fc640 10 monclient: _finish_auth 0 2023-12-08T00:18:54.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc33e7fc640 10 monclient: _check_auth_tickets 2023-12-08T00:18:54.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc33e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:54.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc345d47640 5 monclient: authenticate success, global_id 15816 2023-12-08T00:18:54.198 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.195+0000 7fc345d47640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-08T00:18:54.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc33e7fc640 1 -- 172.21.15.129:0/2939742075 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3340319d0 con 0x7fc34013f590 2023-12-08T00:18:54.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc33e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:54.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc33e7fc640 1 -- 172.21.15.129:0/2939742075 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334031d60 con 0x7fc34013f590 2023-12-08T00:18:54.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc33e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc33e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:54.199 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc33e7fc640 10 monclient: dump: 2023-12-08T00:18:54.199 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 4 set_mon_vals no callback set 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 10 monclient: shutdown 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:54.200 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 1 -- 172.21.15.129:0/2939742075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc34013f590 msgr2=0x7fc3401453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 1 --2- 172.21.15.129:0/2939742075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc34013f590 0x7fc3401453f0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7fc34006c610 tx=0x7fc33402b910 comp rx=0 tx=0).stop 2023-12-08T00:18:54.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 1 -- 172.21.15.129:0/2939742075 shutdown_connections 2023-12-08T00:18:54.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 1 --2- 172.21.15.129:0/2939742075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc34013f590 0x7fc3401453f0 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 1 --2- 172.21.15.129:0/2939742075 >> v2:172.21.15.184:3300/0 conn(0x7fc34006bf50 0x7fc34013f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.197+0000 7fc345d47640 1 -- 172.21.15.129:0/2939742075 >> 172.21.15.129:0/2939742075 conn(0x7fc3400b4c20 msgr2=0x7fc34006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:54.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.198+0000 7fc345d47640 1 -- 172.21.15.129:0/2939742075 shutdown_connections 2023-12-08T00:18:54.201 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.198+0000 7fc345d47640 1 -- 172.21.15.129:0/2939742075 wait complete. 2023-12-08T00:18:54.203 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.198+0000 7fc345d47640 10 monclient: build_initial_monmap 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.198+0000 7fc345d47640 10 monclient: monmap: 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.198+0000 7fc345d47640 1 librados: starting msgr at 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 librados: starting objecter 2023-12-08T00:18:54.204 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 Processor -- start 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 -- start start 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 librados: setting wanted keys 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 librados: calling monclient init 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient: init 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc34006bf50 0x7fc340140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): picked mon.noname-b con 0x7fc34006bf50 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc34013f590 0x7fc340140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): picked mon.noname-c con 0x7fc34013f590 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc340145930 0x7fc340143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): picked mon.noname-a con 0x7fc340145930 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3401423f0 con 0x7fc340145930 2023-12-08T00:18:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc340143630 con 0x7fc34006bf50 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3401425f0 con 0x7fc34013f590 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.199+0000 7fc345d47640 10 monclient(hunting): authenticate will time out at 3334.386875s 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc340145930 0x7fc340143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc340145930 0x7fc340143ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50138/0 (socket says 172.21.15.129:50138) 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33ffff640 1 -- 172.21.15.129:0/369788151 learned_addr learned my addr 172.21.15.129:0/369788151 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33ffff640 10 monclient(hunting): get_auth_request con 0x7fc340145930 auth_method 0 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33ffff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33effd640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc34013f590 0x7fc340140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.206 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33effd640 10 monclient(hunting): get_auth_request con 0x7fc34013f590 auth_method 0 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33effd640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.200+0000 7fc33effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.201+0000 7fc33ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.201+0000 7fc33ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.201+0000 7fc33ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.201+0000 7fc33effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.201+0000 7fc33effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.201+0000 7fc33effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.202+0000 7fc33f7fe640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc34006bf50 0x7fc340140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.202+0000 7fc33f7fe640 10 monclient(hunting): get_auth_request con 0x7fc34006bf50 auth_method 0 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.202+0000 7fc33f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.202+0000 7fc33f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.207 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.202+0000 7fc33f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.208 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.203+0000 7fc33f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.203+0000 7fc33f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.203+0000 7fc33f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.203+0000 7fc33ffff640 10 monclient(hunting): handle_auth_done global_id 15819 payload 824 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 1 -- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc34013f590 msgr2=0x7fc340140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc34013f590 0x7fc340140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 1 -- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc34006bf50 msgr2=0x7fc340140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc34006bf50 0x7fc340140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 1 monclient: found mon.noname-a 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.204+0000 7fc33ffff640 1 -- 172.21.15.129:0/369788151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340094d0 con 0x7fc340145930 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.206+0000 7fc33ffff640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc340145930 0x7fc340143ac0 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7fc33800c6b0 tx=0x7fc33800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.206+0000 7fc327fff640 10 client.?.objecter ms_handle_connect 0x7fc340145930 2023-12-08T00:18:54.209 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.206+0000 7fc327fff640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:54.211 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 1 -- 172.21.15.129:0/369788151 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc338003b70 con 0x7fc340145930 2023-12-08T00:18:54.211 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.211 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:54.211 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: dump: 2023-12-08T00:18:54.211 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.211 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: _finish_auth 0 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: _check_auth_tickets 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 1 -- 172.21.15.129:0/369788151 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc338003cf0 con 0x7fc340145930 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 1 -- 172.21.15.129:0/369788151 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc338003ea0 con 0x7fc340145930 2023-12-08T00:18:54.212 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc327fff640 10 monclient: dump: 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 5 monclient: authenticate success, global_id 15819 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc33dffb640 4 set_mon_vals no callback set 2023-12-08T00:18:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 10 monclient: _renew_subs 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3401428d0 con 0x7fc340145930 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 10 client.15819.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 10 monclient: _renew_subs 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3401441e0 con 0x7fc340145930 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.209+0000 7fc345d47640 1 librados: init done 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.210+0000 7fc345d47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.210+0000 7fc345d47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:54.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.210+0000 7fc345d47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.215 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.210+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3400aead0 con 0x7fc340145930 2023-12-08T00:18:54.231 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.230+0000 7fc327fff640 1 -- 172.21.15.129:0/369788151 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc338005310 con 0x7fc340145930 2023-12-08T00:18:54.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.230+0000 7fc327fff640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc32c06c870 0x7fc32c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.231+0000 7fc33f7fe640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc32c06c870 0x7fc32c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.231+0000 7fc33f7fe640 10 monclient: get_auth_request con 0x7fc32c06c870 auth_method 0 2023-12-08T00:18:54.232 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.231+0000 7fc327fff640 10 client.15819.objecter ms_handle_connect 0x7fc32c06c870 2023-12-08T00:18:54.233 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.231+0000 7fc33f7fe640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc32c06c870 0x7fc32c06ed30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc330009b40 tx=0x7fc330009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 1 -- 172.21.15.129:0/369788151 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1049..1049 src has 999..1049) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fc3380589a0 con 0x7fc340145930 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 10 client.15819.objecter ms_dispatch 0x7fc340000b90 osd_map(1049..1049 src has 999..1049) v4 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 3 client.15819.objecter handle_osd_map got epochs [1049,1049] > 0 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 3 client.15819.objecter handle_osd_map decoding full epoch 1049 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 20 client.15819.objecter dump_active .. 0 homeless 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 1 -- 172.21.15.129:0/369788151 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc338054ce0 con 0x7fc340145930 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:54.464 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.462+0000 7fc327fff640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:54.659 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.658+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 --> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc34013f9d0 con 0x7fc32c06c870 2023-12-08T00:18:54.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.662+0000 7fc327fff640 1 -- 172.21.15.129:0/369788151 <== mgr.4114 v2:172.21.15.184:6800/2561365197 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151935 (secure 0 0 0) 0x7fc330011a60 con 0x7fc32c06c870 2023-12-08T00:18:54.665 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-12-08T00:18:54.666 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 10 librados: watch_flush enter 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 10 librados: watch_flush exit 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 20 client.15819.objecter shutdown clearing up homeless session... 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 10 client.15819.objecter successfully canceled tick 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc32c06c870 msgr2=0x7fc32c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc32c06c870 0x7fc32c06ed30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc330009b40 tx=0x7fc330009190 comp rx=0 tx=0).stop 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 10 monclient: shutdown 2023-12-08T00:18:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc340145930 msgr2=0x7fc340143ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc340145930 0x7fc340143ac0 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7fc33800c6b0 tx=0x7fc33800cb70 comp rx=0 tx=0).stop 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 shutdown_connections 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7fc32c06c870 0x7fc32c06ed30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc340145930 0x7fc340143ac0 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc34013f590 0x7fc340140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 --2- 172.21.15.129:0/369788151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc34006bf50 0x7fc340140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.665+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 >> 172.21.15.129:0/369788151 conn(0x7fc3400b4c20 msgr2=0x7fc34013cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.666+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 shutdown_connections 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.666+0000 7fc345d47640 1 -- 172.21.15.129:0/369788151 wait complete. 2023-12-08T00:18:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.666+0000 7fc345d47640 1 librados: shutdown 2023-12-08T00:18:54.679 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":2039,"stamp":"2023-12-08T00:18:54.146900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":97,"num_write":134,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":583476,"kb_used_data":45300,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754391244,"statfs":{"total":773094113280,"available":772496633856,"internally_reserved":0,"allocated":46387200,"data_stored":21850546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002902"},"pg_stats":[{"pgid":"15.1f","version":"0'0","reported_seq":1014,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.928334+0000","last_change":"2023-12-08T00:18:18.928334+0000","last_active":"2023-12-08T00:18:18.928334+0000","last_peered":"2023-12-08T00:18:18.928334+0000","last_clean":"2023-12-08T00:18:18.928334+0000","last_became_active":"2023-12-07T23:47:23.187382+0000","last_became_peered":"2023-12-07T23:47:23.187382+0000","last_unstale":"2023-12-08T00:18:18.928334+0000","last_undegraded":"2023-12-08T00:18:18.928334+0000","last_fullsized":"2023-12-08T00:18:18.928334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.926408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.926408+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.926408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:43:17.526124+0000","scrub_duration":0.071049899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.016342+0000","last_change":"2023-12-08T00:18:21.017002+0000","last_active":"2023-12-08T00:18:21.016342+0000","last_peered":"2023-12-08T00:18:21.016342+0000","last_clean":"2023-12-08T00:18:21.016342+0000","last_became_active":"2023-12-07T23:47:26.088721+0000","last_became_peered":"2023-12-07T23:47:26.088721+0000","last_unstale":"2023-12-08T00:18:21.016342+0000","last_undegraded":"2023-12-08T00:18:21.016342+0000","last_fullsized":"2023-12-08T00:18:21.016342+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.014984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.014984+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.014984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T12:16:56.647346+0000","scrub_duration":0.02649731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1025,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.568533+0000","last_change":"2023-12-08T00:18:10.233299+0000","last_active":"2023-12-08T00:18:11.568533+0000","last_peered":"2023-12-08T00:18:11.568533+0000","last_clean":"2023-12-08T00:18:11.568533+0000","last_became_active":"2023-12-07T23:47:18.658074+0000","last_became_peered":"2023-12-07T23:47:18.658074+0000","last_unstale":"2023-12-08T00:18:11.568533+0000","last_undegraded":"2023-12-08T00:18:11.568533+0000","last_fullsized":"2023-12-08T00:18:11.568533+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:10.231235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:10.231235+0000","last_clean_scrub_stamp":"2023-12-08T00:18:10.231235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:33:04.158595+0000","scrub_duration":0.060972367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1000,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:15.211477+0000","last_change":"2023-12-08T00:18:15.211477+0000","last_active":"2023-12-08T00:18:15.211477+0000","last_peered":"2023-12-08T00:18:15.211477+0000","last_clean":"2023-12-08T00:18:15.211477+0000","last_became_active":"2023-12-07T23:47:39.133797+0000","last_became_peered":"2023-12-07T23:47:39.133797+0000","last_unstale":"2023-12-08T00:18:15.211477+0000","last_undegraded":"2023-12-08T00:18:15.211477+0000","last_fullsized":"2023-12-08T00:18:15.211477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:15.209401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:15.209401+0000","last_clean_scrub_stamp":"2023-12-08T00:18:15.209401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:18:52.612018+0000","scrub_duration":0.040096010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1007,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:32.801434+0000","last_change":"2023-12-08T00:18:32.801434+0000","last_active":"2023-12-08T00:18:32.801434+0000","last_peered":"2023-12-08T00:18:32.801434+0000","last_clean":"2023-12-08T00:18:32.801434+0000","last_became_active":"2023-12-07T23:47:20.132690+0000","last_became_peered":"2023-12-07T23:47:20.132690+0000","last_unstale":"2023-12-08T00:18:32.801434+0000","last_undegraded":"2023-12-08T00:18:32.801434+0000","last_fullsized":"2023-12-08T00:18:32.801434+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:32.799312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:32.799312+0000","last_clean_scrub_stamp":"2023-12-08T00:18:32.799312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:30:21.468519+0000","scrub_duration":0.076036618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.267961+0000","last_change":"2023-12-08T00:18:14.984114+0000","last_active":"2023-12-08T00:18:16.267961+0000","last_peered":"2023-12-08T00:18:16.267961+0000","last_clean":"2023-12-08T00:18:16.267961+0000","last_became_active":"2023-12-07T23:47:18.926939+0000","last_became_peered":"2023-12-07T23:47:18.926939+0000","last_unstale":"2023-12-08T00:18:16.267961+0000","last_undegraded":"2023-12-08T00:18:16.267961+0000","last_fullsized":"2023-12-08T00:18:16.267961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:14.982044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:14.982044+0000","last_clean_scrub_stamp":"2023-12-08T00:18:14.982044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:53:58.858191+0000","scrub_duration":0.029090370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1","version":"1048'8","reported_seq":1021,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.206631+0000","last_change":"2023-12-08T00:18:14.899000+0000","last_active":"2023-12-08T00:18:16.206631+0000","last_peered":"2023-12-08T00:18:16.206631+0000","last_clean":"2023-12-08T00:18:16.206631+0000","last_became_active":"2023-12-07T23:47:15.386185+0000","last_became_peered":"2023-12-07T23:47:15.386185+0000","last_unstale":"2023-12-08T00:18:16.206631+0000","last_undegraded":"2023-12-08T00:18:16.206631+0000","last_fullsized":"2023-12-08T00:18:16.206631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'8","last_scrub_stamp":"2023-12-08T00:18:14.896730+0000","last_deep_scrub":"1048'8","last_deep_scrub_stamp":"2023-12-08T00:18:14.896730+0000","last_clean_scrub_stamp":"2023-12-08T00:18:14.896730+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:26:28.033722+0000","scrub_duration":0.071940854999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.991668+0000","last_change":"2023-12-08T00:18:18.992364+0000","last_active":"2023-12-08T00:18:18.991668+0000","last_peered":"2023-12-08T00:18:18.991668+0000","last_clean":"2023-12-08T00:18:18.991668+0000","last_became_active":"2023-12-07T23:47:18.700591+0000","last_became_peered":"2023-12-07T23:47:18.700591+0000","last_unstale":"2023-12-08T00:18:18.991668+0000","last_undegraded":"2023-12-08T00:18:18.991668+0000","last_fullsized":"2023-12-08T00:18:18.991668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.990338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.990338+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.990338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:58:57.319194+0000","scrub_duration":0.025705731999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:23.067992+0000","last_change":"2023-12-08T00:18:23.069096+0000","last_active":"2023-12-08T00:18:23.067992+0000","last_peered":"2023-12-08T00:18:23.067992+0000","last_clean":"2023-12-08T00:18:23.067992+0000","last_became_active":"2023-12-07T23:47:16.101044+0000","last_became_peered":"2023-12-07T23:47:16.101044+0000","last_unstale":"2023-12-08T00:18:23.067992+0000","last_undegraded":"2023-12-08T00:18:23.067992+0000","last_fullsized":"2023-12-08T00:18:23.067992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:23.063901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:23.063901+0000","last_clean_scrub_stamp":"2023-12-08T00:18:23.063901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:07:11.581982+0000","scrub_duration":0.072925316000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:22.000467+0000","last_change":"2023-12-08T00:18:22.001151+0000","last_active":"2023-12-08T00:18:22.000467+0000","last_peered":"2023-12-08T00:18:22.000467+0000","last_clean":"2023-12-08T00:18:22.000467+0000","last_became_active":"2023-12-07T23:47:18.123992+0000","last_became_peered":"2023-12-07T23:47:18.123992+0000","last_unstale":"2023-12-08T00:18:22.000467+0000","last_undegraded":"2023-12-08T00:18:22.000467+0000","last_fullsized":"2023-12-08T00:18:22.000467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.999077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.999077+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.999077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T00:20:22.858450+0000","scrub_duration":0.039753019000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1007,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:33.794962+0000","last_change":"2023-12-08T00:18:33.794962+0000","last_active":"2023-12-08T00:18:33.794962+0000","last_peered":"2023-12-08T00:18:33.794962+0000","last_clean":"2023-12-08T00:18:33.794962+0000","last_became_active":"2023-12-07T23:47:19.507339+0000","last_became_peered":"2023-12-07T23:47:19.507339+0000","last_unstale":"2023-12-08T00:18:33.794962+0000","last_undegraded":"2023-12-08T00:18:33.794962+0000","last_fullsized":"2023-12-08T00:18:33.794962+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:33.792911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:33.792911+0000","last_clean_scrub_stamp":"2023-12-08T00:18:33.792911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:22:08.625795+0000","scrub_duration":0.078319633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1023,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.825887+0000","last_change":"2023-12-08T00:18:24.825887+0000","last_active":"2023-12-08T00:18:24.825887+0000","last_peered":"2023-12-08T00:18:24.825887+0000","last_clean":"2023-12-08T00:18:24.825887+0000","last_became_active":"2023-12-07T23:48:06.627236+0000","last_became_peered":"2023-12-07T23:48:06.627236+0000","last_unstale":"2023-12-08T00:18:24.825887+0000","last_undegraded":"2023-12-08T00:18:24.825887+0000","last_fullsized":"2023-12-08T00:18:24.825887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.823891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.823891+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.823891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T10:14:41.955238+0000","scrub_duration":0.037228697999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1022,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:34.852391+0000","last_change":"2023-12-08T00:18:34.852391+0000","last_active":"2023-12-08T00:18:34.852391+0000","last_peered":"2023-12-08T00:18:34.852391+0000","last_clean":"2023-12-08T00:18:34.852391+0000","last_became_active":"2023-12-07T23:47:18.028936+0000","last_became_peered":"2023-12-07T23:47:18.028936+0000","last_unstale":"2023-12-08T00:18:34.852391+0000","last_undegraded":"2023-12-08T00:18:34.852391+0000","last_fullsized":"2023-12-08T00:18:34.852391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:34.848068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:34.848068+0000","last_clean_scrub_stamp":"2023-12-08T00:18:34.848068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:58:00.640459+0000","scrub_duration":0.048325191000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"1047'90","reported_seq":1135,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.199266+0000","last_change":"2023-12-08T00:18:10.088999+0000","last_active":"2023-12-08T00:18:16.199266+0000","last_peered":"2023-12-08T00:18:16.199266+0000","last_clean":"2023-12-08T00:18:16.199266+0000","last_became_active":"2023-12-07T23:46:05.543764+0000","last_became_peered":"2023-12-07T23:46:05.543764+0000","last_unstale":"2023-12-08T00:18:16.199266+0000","last_undegraded":"2023-12-08T00:18:16.199266+0000","last_fullsized":"2023-12-08T00:18:16.199266+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"1047'90","last_scrub_stamp":"2023-12-08T00:18:10.084655+0000","last_deep_scrub":"1047'90","last_deep_scrub_stamp":"2023-12-08T00:18:10.084655+0000","last_clean_scrub_stamp":"2023-12-08T00:18:10.084655+0000","objects_scrubbed":2,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:51:22.482725+0000","scrub_duration":0.204482201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1011,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:19.754747+0000","last_change":"2023-12-08T00:18:19.755399+0000","last_active":"2023-12-08T00:18:19.754747+0000","last_peered":"2023-12-08T00:18:19.754747+0000","last_clean":"2023-12-08T00:18:19.754747+0000","last_became_active":"2023-12-07T23:46:13.810550+0000","last_became_peered":"2023-12-07T23:46:13.810550+0000","last_unstale":"2023-12-08T00:18:19.754747+0000","last_undegraded":"2023-12-08T00:18:19.754747+0000","last_fullsized":"2023-12-08T00:18:19.754747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:19.752924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:19.752924+0000","last_clean_scrub_stamp":"2023-12-08T00:18:19.752924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T02:35:07.286744+0000","scrub_duration":0.028125618000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1006,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.840013+0000","last_change":"2023-12-08T00:18:21.840013+0000","last_active":"2023-12-08T00:18:21.840013+0000","last_peered":"2023-12-08T00:18:21.840013+0000","last_clean":"2023-12-08T00:18:21.840013+0000","last_became_active":"2023-12-07T23:47:18.104193+0000","last_became_peered":"2023-12-07T23:47:18.104193+0000","last_unstale":"2023-12-08T00:18:21.840013+0000","last_undegraded":"2023-12-08T00:18:21.840013+0000","last_fullsized":"2023-12-08T00:18:21.840013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.837952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.837952+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.837952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:29:43.938898+0000","scrub_duration":0.057958458999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1012,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.711660+0000","last_change":"2023-12-08T00:18:15.661804+0000","last_active":"2023-12-08T00:18:16.711660+0000","last_peered":"2023-12-08T00:18:16.711660+0000","last_clean":"2023-12-08T00:18:16.711660+0000","last_became_active":"2023-12-07T23:46:13.368147+0000","last_became_peered":"2023-12-07T23:46:13.368147+0000","last_unstale":"2023-12-08T00:18:16.711660+0000","last_undegraded":"2023-12-08T00:18:16.711660+0000","last_fullsized":"2023-12-08T00:18:16.711660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:15.659834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:15.659834+0000","last_clean_scrub_stamp":"2023-12-08T00:18:15.659834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:23:11.986021+0000","scrub_duration":0.02440811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1020,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:20.275795+0000","last_change":"2023-12-08T00:18:20.275795+0000","last_active":"2023-12-08T00:18:20.275795+0000","last_peered":"2023-12-08T00:18:20.275795+0000","last_clean":"2023-12-08T00:18:20.275795+0000","last_became_active":"2023-12-07T23:47:25.699009+0000","last_became_peered":"2023-12-07T23:47:25.699009+0000","last_unstale":"2023-12-08T00:18:20.275795+0000","last_undegraded":"2023-12-08T00:18:20.275795+0000","last_fullsized":"2023-12-08T00:18:20.275795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:20.273808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:20.273808+0000","last_clean_scrub_stamp":"2023-12-08T00:18:20.273808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:21:30.745491+0000","scrub_duration":0.022630939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1017,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.950346+0000","last_change":"2023-12-08T00:18:12.950346+0000","last_active":"2023-12-08T00:18:12.950346+0000","last_peered":"2023-12-08T00:18:12.950346+0000","last_clean":"2023-12-08T00:18:12.950346+0000","last_became_active":"2023-12-07T23:46:13.313457+0000","last_became_peered":"2023-12-07T23:46:13.313457+0000","last_unstale":"2023-12-08T00:18:12.950346+0000","last_undegraded":"2023-12-08T00:18:12.950346+0000","last_fullsized":"2023-12-08T00:18:12.950346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.947112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.947112+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.947112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:18:48.425554+0000","scrub_duration":0.031302554000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1018,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.283840+0000","last_change":"2023-12-08T00:18:21.284555+0000","last_active":"2023-12-08T00:18:21.283840+0000","last_peered":"2023-12-08T00:18:21.283840+0000","last_clean":"2023-12-08T00:18:21.283840+0000","last_became_active":"2023-12-07T23:47:26.173821+0000","last_became_peered":"2023-12-07T23:47:26.173821+0000","last_unstale":"2023-12-08T00:18:21.283840+0000","last_undegraded":"2023-12-08T00:18:21.283840+0000","last_fullsized":"2023-12-08T00:18:21.283840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.282454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.282454+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.282454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:51:49.950914+0000","scrub_duration":0.068318492999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1011,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:11.564346+0000","last_change":"2023-12-08T00:18:11.564346+0000","last_active":"2023-12-08T00:18:11.564346+0000","last_peered":"2023-12-08T00:18:11.564346+0000","last_clean":"2023-12-08T00:18:11.564346+0000","last_became_active":"2023-12-07T23:46:13.864841+0000","last_became_peered":"2023-12-07T23:46:13.864841+0000","last_unstale":"2023-12-08T00:18:11.564346+0000","last_undegraded":"2023-12-08T00:18:11.564346+0000","last_fullsized":"2023-12-08T00:18:11.564346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:11.560752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:11.560752+0000","last_clean_scrub_stamp":"2023-12-08T00:18:11.560752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:53:55.150564+0000","scrub_duration":0.046001793999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1020,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.354981+0000","last_change":"2023-12-08T00:18:12.354981+0000","last_active":"2023-12-08T00:18:12.354981+0000","last_peered":"2023-12-08T00:18:12.354981+0000","last_clean":"2023-12-08T00:18:12.354981+0000","last_became_active":"2023-12-07T23:47:46.295177+0000","last_became_peered":"2023-12-07T23:47:46.295177+0000","last_unstale":"2023-12-08T00:18:12.354981+0000","last_undegraded":"2023-12-08T00:18:12.354981+0000","last_fullsized":"2023-12-08T00:18:12.354981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.352990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.352990+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.352990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:12:12.666032+0000","scrub_duration":0.042395521999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1009,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.014895+0000","last_change":"2023-12-08T00:18:10.951427+0000","last_active":"2023-12-08T00:18:12.014895+0000","last_peered":"2023-12-08T00:18:12.014895+0000","last_clean":"2023-12-08T00:18:12.014895+0000","last_became_active":"2023-12-07T23:46:13.877134+0000","last_became_peered":"2023-12-07T23:46:13.877134+0000","last_unstale":"2023-12-08T00:18:12.014895+0000","last_undegraded":"2023-12-08T00:18:12.014895+0000","last_fullsized":"2023-12-08T00:18:12.014895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:10.948269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:10.948269+0000","last_clean_scrub_stamp":"2023-12-08T00:18:10.948269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:22:31.595790+0000","scrub_duration":0.025839298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1002,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.231017+0000","last_change":"2023-12-08T00:18:12.231017+0000","last_active":"2023-12-08T00:18:12.231017+0000","last_peered":"2023-12-08T00:18:12.231017+0000","last_clean":"2023-12-08T00:18:12.231017+0000","last_became_active":"2023-12-07T23:47:18.206551+0000","last_became_peered":"2023-12-07T23:47:18.206551+0000","last_unstale":"2023-12-08T00:18:12.231017+0000","last_undegraded":"2023-12-08T00:18:12.231017+0000","last_fullsized":"2023-12-08T00:18:12.231017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.228656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.228656+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.228656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:03:03.814363+0000","scrub_duration":0.029704253999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1011,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:20.789672+0000","last_change":"2023-12-08T00:18:20.790324+0000","last_active":"2023-12-08T00:18:20.789672+0000","last_peered":"2023-12-08T00:18:20.789672+0000","last_clean":"2023-12-08T00:18:20.789672+0000","last_became_active":"2023-12-07T23:46:13.441740+0000","last_became_peered":"2023-12-07T23:46:13.441740+0000","last_unstale":"2023-12-08T00:18:20.789672+0000","last_undegraded":"2023-12-08T00:18:20.789672+0000","last_fullsized":"2023-12-08T00:18:20.789672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:20.788355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:20.788355+0000","last_clean_scrub_stamp":"2023-12-08T00:18:20.788355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:28:10.740579+0000","scrub_duration":0.026280008000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.263509+0000","last_change":"2023-12-08T00:18:16.061409+0000","last_active":"2023-12-08T00:18:16.263509+0000","last_peered":"2023-12-08T00:18:16.263509+0000","last_clean":"2023-12-08T00:18:16.263509+0000","last_became_active":"2023-12-07T23:47:15.770063+0000","last_became_peered":"2023-12-07T23:47:15.770063+0000","last_unstale":"2023-12-08T00:18:16.263509+0000","last_undegraded":"2023-12-08T00:18:16.263509+0000","last_fullsized":"2023-12-08T00:18:16.263509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:16.059310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:16.059310+0000","last_clean_scrub_stamp":"2023-12-08T00:18:16.059310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:47:14.094698+0000","scrub_duration":0.10337629299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1027,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.844439+0000","last_change":"2023-12-08T00:18:12.844439+0000","last_active":"2023-12-08T00:18:12.844439+0000","last_peered":"2023-12-08T00:18:12.844439+0000","last_clean":"2023-12-08T00:18:12.844439+0000","last_became_active":"2023-12-07T23:46:13.629518+0000","last_became_peered":"2023-12-07T23:46:13.629518+0000","last_unstale":"2023-12-08T00:18:12.844439+0000","last_undegraded":"2023-12-08T00:18:12.844439+0000","last_fullsized":"2023-12-08T00:18:12.844439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:12.842337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:12.842337+0000","last_clean_scrub_stamp":"2023-12-08T00:18:12.842337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:19:54.361371+0000","scrub_duration":0.032484658999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1004,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:12.016563+0000","last_change":"2023-12-08T00:18:11.987531+0000","last_active":"2023-12-08T00:18:12.016563+0000","last_peered":"2023-12-08T00:18:12.016563+0000","last_clean":"2023-12-08T00:18:12.016563+0000","last_became_active":"2023-12-07T23:47:15.693158+0000","last_became_peered":"2023-12-07T23:47:15.693158+0000","last_unstale":"2023-12-08T00:18:12.016563+0000","last_undegraded":"2023-12-08T00:18:12.016563+0000","last_fullsized":"2023-12-08T00:18:12.016563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:11.985590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:11.985590+0000","last_clean_scrub_stamp":"2023-12-08T00:18:11.985590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T08:12:50.655676+0000","scrub_duration":0.041049627999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1011,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.800605+0000","last_change":"2023-12-08T00:18:24.800605+0000","last_active":"2023-12-08T00:18:24.800605+0000","last_peered":"2023-12-08T00:18:24.800605+0000","last_clean":"2023-12-08T00:18:24.800605+0000","last_became_active":"2023-12-07T23:46:13.615074+0000","last_became_peered":"2023-12-07T23:46:13.615074+0000","last_unstale":"2023-12-08T00:18:24.800605+0000","last_undegraded":"2023-12-08T00:18:24.800605+0000","last_fullsized":"2023-12-08T00:18:24.800605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.795744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.795744+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.795744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:11:59.912669+0000","scrub_duration":0.036985285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1012,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.056002+0000","last_change":"2023-12-08T00:18:13.945767+0000","last_active":"2023-12-08T00:18:16.056002+0000","last_peered":"2023-12-08T00:18:16.056002+0000","last_clean":"2023-12-08T00:18:16.056002+0000","last_became_active":"2023-12-07T23:47:21.229799+0000","last_became_peered":"2023-12-07T23:47:21.229799+0000","last_unstale":"2023-12-08T00:18:16.056002+0000","last_undegraded":"2023-12-08T00:18:16.056002+0000","last_fullsized":"2023-12-08T00:18:16.056002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:13.943649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:13.943649+0000","last_clean_scrub_stamp":"2023-12-08T00:18:13.943649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:13:49.022488+0000","scrub_duration":0.047138658999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.773561+0000","last_change":"2023-12-08T00:18:18.773561+0000","last_active":"2023-12-08T00:18:18.773561+0000","last_peered":"2023-12-08T00:18:18.773561+0000","last_clean":"2023-12-08T00:18:18.773561+0000","last_became_active":"2023-12-07T23:47:36.616695+0000","last_became_peered":"2023-12-07T23:47:36.616695+0000","last_unstale":"2023-12-08T00:18:18.773561+0000","last_undegraded":"2023-12-08T00:18:18.773561+0000","last_fullsized":"2023-12-08T00:18:18.773561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.771581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.771581+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.771581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T01:13:54.973664+0000","scrub_duration":0.042405533000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.796315+0000","last_change":"2023-12-08T00:18:24.797325+0000","last_active":"2023-12-08T00:18:24.796315+0000","last_peered":"2023-12-08T00:18:24.796315+0000","last_clean":"2023-12-08T00:18:24.796315+0000","last_became_active":"2023-12-07T23:47:36.567130+0000","last_became_peered":"2023-12-07T23:47:36.567130+0000","last_unstale":"2023-12-08T00:18:24.796315+0000","last_undegraded":"2023-12-08T00:18:24.796315+0000","last_fullsized":"2023-12-08T00:18:24.796315+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.794205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.794205+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.794205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T03:56:08.870545+0000","scrub_duration":0.025459683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1004,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:27.774320+0000","last_change":"2023-12-08T00:18:27.774978+0000","last_active":"2023-12-08T00:18:27.774320+0000","last_peered":"2023-12-08T00:18:27.774320+0000","last_clean":"2023-12-08T00:18:27.774320+0000","last_became_active":"2023-12-07T23:47:37.279080+0000","last_became_peered":"2023-12-07T23:47:37.279080+0000","last_unstale":"2023-12-08T00:18:27.774320+0000","last_undegraded":"2023-12-08T00:18:27.774320+0000","last_fullsized":"2023-12-08T00:18:27.774320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:27.772954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:27.772954+0000","last_clean_scrub_stamp":"2023-12-08T00:18:27.772954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T06:35:04.612518+0000","scrub_duration":0.078081676000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:30.795374+0000","last_change":"2023-12-08T00:18:30.796009+0000","last_active":"2023-12-08T00:18:30.795374+0000","last_peered":"2023-12-08T00:18:30.795374+0000","last_clean":"2023-12-08T00:18:30.795374+0000","last_became_active":"2023-12-07T23:47:21.389382+0000","last_became_peered":"2023-12-07T23:47:21.389382+0000","last_unstale":"2023-12-08T00:18:30.795374+0000","last_undegraded":"2023-12-08T00:18:30.795374+0000","last_fullsized":"2023-12-08T00:18:30.795374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:30.794035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:30.794035+0000","last_clean_scrub_stamp":"2023-12-08T00:18:30.794035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:55:16.876714+0000","scrub_duration":0.022613728999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1019,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:24.193882+0000","last_change":"2023-12-08T00:18:24.193882+0000","last_active":"2023-12-08T00:18:24.193882+0000","last_peered":"2023-12-08T00:18:24.193882+0000","last_clean":"2023-12-08T00:18:24.193882+0000","last_became_active":"2023-12-07T23:47:20.968485+0000","last_became_peered":"2023-12-07T23:47:20.968485+0000","last_unstale":"2023-12-08T00:18:24.193882+0000","last_undegraded":"2023-12-08T00:18:24.193882+0000","last_fullsized":"2023-12-08T00:18:24.193882+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:24.191804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:24.191804+0000","last_clean_scrub_stamp":"2023-12-08T00:18:24.191804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:52:20.846742+0000","scrub_duration":0.047630634999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1001,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:17.191083+0000","last_change":"2023-12-08T00:18:17.191083+0000","last_active":"2023-12-08T00:18:17.191083+0000","last_peered":"2023-12-08T00:18:17.191083+0000","last_clean":"2023-12-08T00:18:17.191083+0000","last_became_active":"2023-12-07T23:47:21.395304+0000","last_became_peered":"2023-12-07T23:47:21.395304+0000","last_unstale":"2023-12-08T00:18:17.191083+0000","last_undegraded":"2023-12-08T00:18:17.191083+0000","last_fullsized":"2023-12-08T00:18:17.191083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:17.185405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:17.185405+0000","last_clean_scrub_stamp":"2023-12-08T00:18:17.185405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T11:07:03.933546+0000","scrub_duration":0.042693994999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1018,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:23.249975+0000","last_change":"2023-12-08T00:18:23.249975+0000","last_active":"2023-12-08T00:18:23.249975+0000","last_peered":"2023-12-08T00:18:23.249975+0000","last_clean":"2023-12-08T00:18:23.249975+0000","last_became_active":"2023-12-07T23:47:38.790323+0000","last_became_peered":"2023-12-07T23:47:38.790323+0000","last_unstale":"2023-12-08T00:18:23.249975+0000","last_undegraded":"2023-12-08T00:18:23.249975+0000","last_fullsized":"2023-12-08T00:18:23.249975+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:23.248033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:23.248033+0000","last_clean_scrub_stamp":"2023-12-08T00:18:23.248033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T05:26:09.453050+0000","scrub_duration":0.072562849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1005,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:26.742425+0000","last_change":"2023-12-08T00:18:26.743113+0000","last_active":"2023-12-08T00:18:26.742425+0000","last_peered":"2023-12-08T00:18:26.742425+0000","last_clean":"2023-12-08T00:18:26.742425+0000","last_became_active":"2023-12-07T23:47:37.180489+0000","last_became_peered":"2023-12-07T23:47:37.180489+0000","last_unstale":"2023-12-08T00:18:26.742425+0000","last_undegraded":"2023-12-08T00:18:26.742425+0000","last_fullsized":"2023-12-08T00:18:26.742425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:26.741029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:26.741029+0000","last_clean_scrub_stamp":"2023-12-08T00:18:26.741029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:52:01.190729+0000","scrub_duration":0.025969953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1021,"reported_epoch":1048,"state":"active+clean","last_fresh":"2023-12-08T00:18:16.812103+0000","last_change":"2023-12-08T00:18:16.812103+0000","last_active":"2023-12-08T00:18:16.812103+0000","last_peered":"2023-12-08T00:18:16.812103+0000","last_clean":"2023-12-08T00:18:16.812103+0000","last_became_active":"2023-12-07T23:48:09.186728+0000","last_became_peered":"2023-12-07T23:48:09.186728+0000","last_unstale":"2023-12-08T00:18:16.812103+0000","last_undegraded":"2023-12-08T00:18:16.812103+0000","last_fullsized":"2023-12-08T00:18:16.812103+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:16.809929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:16.809929+0000","last_clean_scrub_stamp":"2023-12-08T00:18:16.809929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T04:14:52.287877+0000","scrub_duration":0.044075022999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1001,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:18.142254+0000","last_change":"2023-12-08T00:18:18.142932+0000","last_active":"2023-12-08T00:18:18.142254+0000","last_peered":"2023-12-08T00:18:18.142254+0000","last_clean":"2023-12-08T00:18:18.142254+0000","last_became_active":"2023-12-07T23:47:20.123312+0000","last_became_peered":"2023-12-07T23:47:20.123312+0000","last_unstale":"2023-12-08T00:18:18.142254+0000","last_undegraded":"2023-12-08T00:18:18.142254+0000","last_fullsized":"2023-12-08T00:18:18.142254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:18.140805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:18.140805+0000","last_clean_scrub_stamp":"2023-12-08T00:18:18.140805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T09:37:49.858263+0000","scrub_duration":0.035369036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1021,"reported_epoch":1049,"state":"active+clean","last_fresh":"2023-12-08T00:18:21.860338+0000","last_change":"2023-12-08T00:18:21.860992+0000","last_active":"2023-12-08T00:18:21.860338+0000","last_peered":"2023-12-08T00:18:21.860338+0000","last_clean":"2023-12-08T00:18:21.860338+0000","last_became_active":"2023-12-07T23:47:18.946564+0000","last_became_peered":"2023-12-07T23:47:18.946564+0000","last_unstale":"2023-12-08T00:18:21.860338+0000","last_undegraded":"2023-12-08T00:18:21.860338+0000","last_fullsized":"2023-12-08T00:18:21.860338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T00:18:21.858949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T00:18:21.858949+0000","last_clean_scrub_stamp":"2023-12-08T00:18:21.858949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-09T07:22:09.518786+0000","scrub_duration":0.075188711000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":127,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":2,"acting":2,"num_store_stats":2},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477129,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81244,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290596,"statfs":{"total":96636764160,"available":96553570304,"internally_reserved":0,"allocated":5787648,"data_stored":2681864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":12.305999999999999,"15min":16.677},"min":{"1min":5.2380000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"max":{"1min":255.59,"5min":255.59,"15min":272.77100000000002},"last":9.3019999999999996},{"interface":"front","average":{"1min":26.963999999999999,"5min":14.023,"15min":19.472000000000001},"min":{"1min":6.8899999999999997,"5min":4.9900000000000002,"15min":4.9649999999999999},"max":{"1min":224.32400000000001,"5min":224.32400000000001,"15min":288.42000000000002},"last":13.858000000000001}]},{"osd":1,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":21.672000000000001,"5min":14.118,"15min":21.623999999999999},"min":{"1min":8.4749999999999996,"5min":6.2279999999999998,"15min":5.7649999999999997},"max":{"1min":122.941,"5min":122.941,"15min":320.57900000000001},"last":10.321},{"interface":"front","average":{"1min":18.599,"5min":14.475,"15min":23.824000000000002},"min":{"1min":7.9180000000000001,"5min":6.7279999999999998,"15min":5.5670000000000002},"max":{"1min":60.479999999999997,"5min":91.882000000000005,"15min":461.51799999999997},"last":8.2330000000000005}]},{"osd":2,"last update":"Fri Dec 8 00:18:39 2023","interfaces":[{"interface":"back","average":{"1min":16.303000000000001,"5min":11.141,"15min":16.201000000000001},"min":{"1min":6.7969999999999997,"5min":6.0389999999999997,"15min":6.0190000000000001},"max":{"1min":70.909999999999997,"5min":70.909999999999997,"15min":246.47499999999999},"last":12.115},{"interface":"front","average":{"1min":25.562999999999999,"5min":16.675999999999998,"15min":23.437999999999999},"min":{"1min":10.052,"5min":8.7279999999999998,"15min":7.3079999999999998},"max":{"1min":136.61199999999999,"5min":136.61199999999999,"15min":356.02100000000002},"last":19.266999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":27.315000000000001,"5min":17.056000000000001,"15min":20.420999999999999},"min":{"1min":11.606,"5min":7.048,"15min":6.7690000000000001},"max":{"1min":160.06100000000001,"5min":160.06100000000001,"15min":180.76400000000001},"last":14.507999999999999},{"interface":"front","average":{"1min":29.664000000000001,"5min":17.638999999999999,"15min":23.712},"min":{"1min":11.324,"5min":8.3040000000000003,"15min":6.2990000000000004},"max":{"1min":173.029,"5min":173.029,"15min":223.91300000000001},"last":10.994999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":34.090000000000003,"5min":18.847000000000001,"15min":25.256},"min":{"1min":12.975,"5min":8.9920000000000009,"15min":7.2889999999999997},"max":{"1min":219.87899999999999,"5min":219.87899999999999,"15min":251.892},"last":12.888999999999999},{"interface":"front","average":{"1min":39.415999999999997,"5min":24.015000000000001,"15min":29.675999999999998},"min":{"1min":14.428000000000001,"5min":11.711,"15min":10.568},"max":{"1min":259.93099999999998,"5min":265.69799999999998,"15min":279.791},"last":15.904999999999999}]},{"osd":6,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":26.574000000000002,"5min":17.271000000000001,"15min":20.309000000000001},"min":{"1min":8.8170000000000002,"5min":7.2850000000000001,"15min":7.2850000000000001},"max":{"1min":162.42699999999999,"5min":214.774,"15min":214.774},"last":18.231000000000002},{"interface":"front","average":{"1min":31.245000000000001,"5min":21.231999999999999,"15min":25.077999999999999},"min":{"1min":11.41,"5min":10.428000000000001,"15min":8.1440000000000001},"max":{"1min":169.095,"5min":229.54300000000001,"15min":229.54300000000001},"last":20.126000000000001}]},{"osd":7,"last update":"Fri Dec 8 00:18:17 2023","interfaces":[{"interface":"back","average":{"1min":38.497,"5min":21.491,"15min":29.875},"min":{"1min":14.987,"5min":8.923,"15min":8.923},"max":{"1min":246.601,"5min":246.601,"15min":281.31},"last":17.417000000000002},{"interface":"front","average":{"1min":30.427,"5min":17.623000000000001,"15min":23.940000000000001},"min":{"1min":12.416,"5min":8.8010000000000002,"15min":7.649},"max":{"1min":163.34700000000001,"5min":163.34700000000001,"15min":255.72499999999999},"last":15.286}]}]},{"osd":2,"up_from":16,"seq":68719477130,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68528,"kb_used_data":5416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303312,"statfs":{"total":96636764160,"available":96566591488,"internally_reserved":0,"allocated":5545984,"data_stored":2548512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":11.218,"5min":9.1470000000000002,"15min":14.103},"min":{"1min":5.5700000000000003,"5min":4.1429999999999998,"15min":4.1429999999999998},"max":{"1min":41.286000000000001,"5min":41.286000000000001,"15min":249.42699999999999},"last":6.8799999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":10.103999999999999,"15min":15.738},"min":{"1min":5.5019999999999998,"5min":5.5019999999999998,"15min":5.1310000000000002},"max":{"1min":42.930999999999997,"5min":42.930999999999997,"15min":241.45599999999999},"last":12.09}]},{"osd":1,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":17.216999999999999,"5min":13.097,"15min":18.321000000000002},"min":{"1min":6.6020000000000003,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":87.658000000000001,"5min":87.658000000000001,"15min":234.279},"last":14.75},{"interface":"front","average":{"1min":14.349,"5min":11.438000000000001,"15min":16.529},"min":{"1min":6.0640000000000001,"5min":6.0640000000000001,"15min":5.9109999999999996},"max":{"1min":82.905000000000001,"5min":82.905000000000001,"15min":258.61599999999999},"last":10.317}]},{"osd":3,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":13.989000000000001,"5min":11.375999999999999,"15min":18.414999999999999},"min":{"1min":6.8099999999999996,"5min":6.2770000000000001,"15min":6.1550000000000002},"max":{"1min":31.184000000000001,"5min":31.184000000000001,"15min":287.56299999999999},"last":16.102},{"interface":"front","average":{"1min":15.461,"5min":14.363,"15min":20.439},"min":{"1min":8.6760000000000002,"5min":8.3059999999999992,"15min":7.149},"max":{"1min":36.813000000000002,"5min":36.813000000000002,"15min":223.20400000000001},"last":12.911}]},{"osd":4,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":18.015999999999998,"5min":15.794,"15min":21.940000000000001},"min":{"1min":11.374000000000001,"5min":8.9220000000000006,"15min":8.1539999999999999},"max":{"1min":40.554000000000002,"5min":40.554000000000002,"15min":213.88},"last":16.789000000000001},{"interface":"front","average":{"1min":13.412000000000001,"5min":12.475,"15min":18.739999999999998},"min":{"1min":8.3490000000000002,"5min":6.7450000000000001,"15min":6.2649999999999997},"max":{"1min":35.106999999999999,"5min":35.106999999999999,"15min":212.494},"last":11.763999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":21.234000000000002,"5min":17.533999999999999,"15min":22.791},"min":{"1min":12.539999999999999,"5min":9.0410000000000004,"15min":8.5470000000000006},"max":{"1min":88.070999999999998,"5min":88.070999999999998,"15min":204.60599999999999},"last":19.213000000000001},{"interface":"front","average":{"1min":17.984000000000002,"5min":14.983000000000001,"15min":19.736999999999998},"min":{"1min":9.3889999999999993,"5min":8.7360000000000007,"15min":7.0039999999999996},"max":{"1min":96.875,"5min":96.875,"15min":248.63499999999999},"last":11.055}]},{"osd":6,"last update":"Fri Dec 8 00:18:21 2023","interfaces":[{"interface":"back","average":{"1min":17.937999999999999,"5min":16.193999999999999,"15min":19.606999999999999},"min":{"1min":10.353999999999999,"5min":10.353999999999999,"15min":7.4240000000000004},"max":{"1min":41.552999999999997,"5min":41.552999999999997,"15min":232.48099999999999},"last":15.416},{"interface":"front","average":{"1min":17.151,"5min":13.273,"15min":19.757000000000001},"min":{"1min":8.2100000000000009,"5min":7.423,"15min":7.423},"max":{"1min":35.909999999999997,"5min":35.909999999999997,"15min":231.422},"last":20.222999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:18:13 2023","interfaces":[{"interface":"back","average":{"1min":17.989000000000001,"5min":14.842000000000001,"15min":23.449999999999999},"min":{"1min":12.250999999999999,"5min":8.6010000000000009,"15min":8.6010000000000009},"max":{"1min":33.475999999999999,"5min":33.475999999999999,"15min":350.52199999999999},"last":21.065999999999999},{"interface":"front","average":{"1min":17.739000000000001,"5min":17.244,"15min":24.251000000000001},"min":{"1min":11.827,"5min":11.411,"15min":10.475},"max":{"1min":38.564999999999998,"5min":38.564999999999998,"15min":235.19200000000001},"last":17.463999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477129,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68764,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303076,"statfs":{"total":96636764160,"available":96566349824,"internally_reserved":0,"allocated":5787648,"data_stored":2681864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 8 00:18:32 2023","interfaces":[{"interface":"back","average":{"1min":9.3889999999999993,"5min":9.7420000000000009,"15min":18.280999999999999},"min":{"1min":4.8860000000000001,"5min":4.827,"15min":4.5830000000000002},"max":{"1min":14.888999999999999,"5min":38.834000000000003,"15min":317.99599999999998},"last":12.744},{"interface":"front","average":{"1min":11.938000000000001,"5min":9.9570000000000007,"15min":17.164000000000001},"min":{"1min":5.5430000000000001,"5min":4.9139999999999997,"15min":4.9139999999999997},"max":{"1min":29.773,"5min":49.262,"15min":313.14400000000001},"last":9.0020000000000007}]},{"osd":2,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":18.568000000000001,"5min":17.286000000000001,"15min":20.753},"min":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":118.601,"5min":196.96700000000001,"15min":228.02500000000001},"last":7.8559999999999999},{"interface":"front","average":{"1min":17.353999999999999,"5min":15.314,"15min":18.536999999999999},"min":{"1min":7.6559999999999997,"5min":5.7530000000000001,"15min":5.6059999999999999},"max":{"1min":43.734000000000002,"5min":168.91300000000001,"15min":197.78899999999999},"last":13.57}]},{"osd":3,"last update":"Fri Dec 8 00:18:11 2023","interfaces":[{"interface":"back","average":{"1min":18.135999999999999,"5min":14.326000000000001,"15min":22.596},"min":{"1min":8.6419999999999995,"5min":8.0510000000000002,"15min":7.0069999999999997},"max":{"1min":115.229,"5min":115.229,"15min":239.50999999999999},"last":12.170999999999999},{"interface":"front","average":{"1min":17.655999999999999,"5min":16.533999999999999,"15min":23.977},"min":{"1min":12.94,"5min":10.667,"15min":9.141},"max":{"1min":34.673000000000002,"5min":34.673000000000002,"15min":241.839},"last":18.196999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:18:11 2023","interfaces":[{"interface":"back","average":{"1min":12.961,"5min":9.7360000000000007,"15min":14.757},"min":{"1min":6.7119999999999997,"5min":5.5570000000000004,"15min":5.5430000000000001},"max":{"1min":41.896999999999998,"5min":41.896999999999998,"15min":211.452},"last":10.114000000000001},{"interface":"front","average":{"1min":18.591000000000001,"5min":17.117999999999999,"15min":22.282},"min":{"1min":11.193,"5min":10.212999999999999,"15min":9.3030000000000008},"max":{"1min":37.057000000000002,"5min":37.057000000000002,"15min":212.459},"last":20.463000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":17.085000000000001,"5min":15.750999999999999,"15min":19.074000000000002},"min":{"1min":9.8079999999999998,"5min":7.8970000000000002,"15min":6.2560000000000002},"max":{"1min":33.323999999999998,"5min":175.68100000000001,"15min":175.68100000000001},"last":14.273999999999999},{"interface":"front","average":{"1min":12.287000000000001,"5min":13.906000000000001,"15min":19.5},"min":{"1min":8.6560000000000006,"5min":6.7789999999999999,"15min":6.6660000000000004},"max":{"1min":31.010000000000002,"5min":186.625,"15min":186.625},"last":10.803000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":17.625,"5min":17.896999999999998,"15min":25.821000000000002},"min":{"1min":11.704000000000001,"5min":11.406000000000001,"15min":11.166},"max":{"1min":28.852,"5min":30.727,"15min":227.892},"last":19.391999999999999},{"interface":"front","average":{"1min":16.443999999999999,"5min":14.975,"15min":20.138000000000002},"min":{"1min":9.5009999999999994,"5min":8.2919999999999998,"15min":8.2919999999999998},"max":{"1min":32.412999999999997,"5min":44.432000000000002,"15min":153.59700000000001},"last":15.513999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:18:22 2023","interfaces":[{"interface":"back","average":{"1min":20.614000000000001,"5min":17.145,"15min":24.812000000000001},"min":{"1min":13.722,"5min":9.3550000000000004,"15min":8.9369999999999994},"max":{"1min":49.183,"5min":49.183,"15min":264.44600000000003},"last":14.907},{"interface":"front","average":{"1min":13.4,"5min":13.119,"15min":20.073},"min":{"1min":8.6920000000000002,"5min":7.5350000000000001,"15min":7.5},"max":{"1min":27.434000000000001,"5min":41.091000000000001,"15min":243.84899999999999},"last":11.417}]}]},{"osd":1,"up_from":14,"seq":60129542535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68924,"kb_used_data":5812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302916,"statfs":{"total":96636764160,"available":96566185984,"internally_reserved":0,"allocated":5951488,"data_stored":2946357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:16 2023","interfaces":[{"interface":"back","average":{"1min":8.8390000000000004,"5min":9.2059999999999995,"15min":16.134},"min":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.2539999999999996},"max":{"1min":14.042999999999999,"5min":18.986999999999998,"15min":324.46600000000001},"last":9.827},{"interface":"front","average":{"1min":12.923999999999999,"5min":13.388999999999999,"15min":21.181999999999999},"min":{"1min":7.9139999999999997,"5min":6.4950000000000001,"15min":5.5369999999999999},"max":{"1min":31.443000000000001,"5min":31.443000000000001,"15min":342.39299999999997},"last":16.559999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:18:16 2023","interfaces":[{"interface":"back","average":{"1min":18.105,"5min":11.507,"15min":19.349},"min":{"1min":6.3029999999999999,"5min":4.7409999999999997,"15min":4.7409999999999997},"max":{"1min":109.697,"5min":109.697,"15min":288.863},"last":6.6280000000000001},{"interface":"front","average":{"1min":32.765999999999998,"5min":17.178999999999998,"15min":24.306000000000001},"min":{"1min":10.532999999999999,"5min":6.9100000000000001,"15min":6.9100000000000001},"max":{"1min":330.75900000000001,"5min":330.75900000000001,"15min":343.49900000000002},"last":14.839}]},{"osd":3,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":13.263,"5min":14.33,"15min":22.765000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"max":{"1min":17.984999999999999,"5min":20.395,"15min":345.07600000000002},"last":17.724},{"interface":"front","average":{"1min":10.297000000000001,"5min":9.077,"15min":15.353},"min":{"1min":7.5209999999999999,"5min":6.1050000000000004,"15min":5.3090000000000002},"max":{"1min":15.089,"5min":16.981999999999999,"15min":174.20599999999999},"last":10.420999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:18:16 2023","interfaces":[{"interface":"back","average":{"1min":13.456,"5min":11.994,"15min":18.782},"min":{"1min":6.8499999999999996,"5min":6.1970000000000001,"15min":6.0419999999999998},"max":{"1min":32.310000000000002,"5min":32.310000000000002,"15min":290.358},"last":7.9279999999999999},{"interface":"front","average":{"1min":12.930999999999999,"5min":12.369,"15min":18.876000000000001},"min":{"1min":8.3000000000000007,"5min":6.1909999999999998,"15min":6.0469999999999997},"max":{"1min":20.007000000000001,"5min":20.007000000000001,"15min":169.90799999999999},"last":12.529999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":32.521999999999998,"5min":19.515000000000001,"15min":24.640999999999998},"min":{"1min":10.952999999999999,"5min":10.938000000000001,"15min":7.7690000000000001},"max":{"1min":333.11799999999999,"5min":333.11799999999999,"15min":371.50599999999997},"last":18.600999999999999},{"interface":"front","average":{"1min":28.724,"5min":15.845000000000001,"15min":19.704000000000001},"min":{"1min":9.0389999999999997,"5min":7.6150000000000002,"15min":7.282},"max":{"1min":298.12099999999998,"5min":298.12099999999998,"15min":298.12099999999998},"last":11.77}]},{"osd":6,"last update":"Fri Dec 8 00:17:59 2023","interfaces":[{"interface":"back","average":{"1min":14.052,"5min":13.733000000000001,"15min":19.829000000000001},"min":{"1min":9.0879999999999992,"5min":7.3490000000000002,"15min":7.3490000000000002},"max":{"1min":19.085000000000001,"5min":24.93,"15min":292.72300000000001},"last":11.034000000000001},{"interface":"front","average":{"1min":15.968999999999999,"5min":14.984,"15min":22.007999999999999},"min":{"1min":10.695,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":18.954999999999998,"5min":26.942,"15min":173.17599999999999},"last":14.459}]},{"osd":7,"last update":"Fri Dec 8 00:18:04 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":16.228999999999999,"15min":26.417000000000002},"min":{"1min":10.32,"5min":9.3789999999999996,"15min":9.3789999999999996},"max":{"1min":94.143000000000001,"5min":94.143000000000001,"15min":350.50999999999999},"last":8.9749999999999996},{"interface":"front","average":{"1min":24.823,"5min":18.376999999999999,"15min":24.126000000000001},"min":{"1min":12.754,"5min":12.005000000000001,"15min":8.3659999999999997},"max":{"1min":90.555999999999997,"5min":90.555999999999997,"15min":304.64499999999998},"last":15.930999999999999}]}]},{"osd":7,"up_from":13,"seq":55834575243,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68528,"kb_used_data":5416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303312,"statfs":{"total":96636764160,"available":96566591488,"internally_reserved":0,"allocated":5545984,"data_stored":2548512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":10.093999999999999,"15min":19.027000000000001},"min":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":4.5179999999999998},"max":{"1min":73.165999999999997,"5min":73.165999999999997,"15min":259.01499999999999},"last":13.07},{"interface":"front","average":{"1min":16.966000000000001,"5min":14.547000000000001,"15min":23.523},"min":{"1min":9.3879999999999999,"5min":6.1369999999999996,"15min":6.016},"max":{"1min":73.781000000000006,"5min":73.781000000000006,"15min":292.73899999999998},"last":14.868}]},{"osd":1,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":11.380000000000001,"5min":10.147,"15min":19.216000000000001},"min":{"1min":4.3289999999999997,"5min":4.3289999999999997,"15min":4.3289999999999997},"max":{"1min":28.245000000000001,"5min":28.245000000000001,"15min":414.57100000000003},"last":10.249000000000001},{"interface":"front","average":{"1min":14.326000000000001,"5min":11.039,"15min":17.794},"min":{"1min":6.327,"5min":6.327,"15min":5.984},"max":{"1min":38.805,"5min":38.805,"15min":333.53800000000001},"last":8.2650000000000006}]},{"osd":2,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":18.044,"5min":13.077,"15min":21.619},"min":{"1min":7.3449999999999998,"5min":6.75,"15min":6.75},"max":{"1min":94.254000000000005,"5min":94.254000000000005,"15min":286.63499999999999},"last":8.9870000000000001},{"interface":"front","average":{"1min":22.062000000000001,"5min":14.420999999999999,"15min":21.707999999999998},"min":{"1min":11.499000000000001,"5min":7.0540000000000003,"15min":6.1280000000000001},"max":{"1min":96.084000000000003,"5min":96.084000000000003,"15min":299.54300000000001},"last":11.865}]},{"osd":3,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":15.101000000000001,"5min":13.878,"15min":23.503},"min":{"1min":7.819,"5min":7.0810000000000004,"15min":6.6289999999999996},"max":{"1min":37.604999999999997,"5min":37.604999999999997,"15min":391.23200000000003},"last":16.399999999999999},{"interface":"front","average":{"1min":17.687000000000001,"5min":16.838000000000001,"15min":25.504000000000001},"min":{"1min":11.420999999999999,"5min":10.843,"15min":8.3949999999999996},"max":{"1min":27.238,"5min":27.238,"15min":362.20299999999997},"last":18.998000000000001}]},{"osd":4,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":17.986000000000001,"5min":17.036999999999999,"15min":23.123999999999999},"min":{"1min":9.2989999999999995,"5min":7.6710000000000003,"15min":7.6710000000000003},"max":{"1min":31.456,"5min":31.456,"15min":293.358},"last":17.189},{"interface":"front","average":{"1min":17.670000000000002,"5min":14.42,"15min":20.253},"min":{"1min":7.6959999999999997,"5min":7.6959999999999997,"15min":6.6769999999999996},"max":{"1min":38.436,"5min":38.436,"15min":275.31799999999998},"last":13.194000000000001}]},{"osd":5,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":22.408000000000001,"5min":18.739999999999998,"15min":26.800999999999998},"min":{"1min":13.476000000000001,"5min":9.6720000000000006,"15min":7.7380000000000004},"max":{"1min":97.998000000000005,"5min":97.998000000000005,"15min":294.56400000000002},"last":19.759},{"interface":"front","average":{"1min":17.391999999999999,"5min":12.614000000000001,"15min":23.106000000000002},"min":{"1min":7.1719999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"max":{"1min":143.78100000000001,"5min":143.78100000000001,"15min":280.83600000000001},"last":18.079999999999998}]},{"osd":6,"last update":"Fri Dec 8 00:18:14 2023","interfaces":[{"interface":"back","average":{"1min":18.853000000000002,"5min":16.222999999999999,"15min":24.637},"min":{"1min":13.026999999999999,"5min":10.958,"15min":8.5579999999999998},"max":{"1min":40.783999999999999,"5min":40.783999999999999,"15min":289.798},"last":10.914},{"interface":"front","average":{"1min":15.396000000000001,"5min":14.058,"15min":23.053000000000001},"min":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"max":{"1min":40.825000000000003,"5min":40.825000000000003,"15min":303.30900000000003},"last":14.467000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575241,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85408,"kb_used_data":5656,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94286432,"statfs":{"total":96636764160,"available":96549306368,"internally_reserved":0,"allocated":5791744,"data_stored":2681869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:20 2023","interfaces":[{"interface":"back","average":{"1min":10.545,"5min":10.394,"15min":14.253},"min":{"1min":4.2409999999999997,"5min":4.2409999999999997,"15min":4.1920000000000002},"max":{"1min":23.716999999999999,"5min":23.716999999999999,"15min":168.34100000000001},"last":10.413},{"interface":"front","average":{"1min":8.9920000000000009,"5min":8.8610000000000007,"15min":13.792},"min":{"1min":4.3869999999999996,"5min":4.3869999999999996,"15min":4.3869999999999996},"max":{"1min":27.768000000000001,"5min":27.768000000000001,"15min":172.452},"last":14.137}]},{"osd":1,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":12.702999999999999,"5min":13.539,"15min":16.152000000000001},"min":{"1min":6.6180000000000003,"5min":6.6180000000000003,"15min":6.1870000000000003},"max":{"1min":18.241,"5min":20.835000000000001,"15min":206.43899999999999},"last":12.071999999999999},{"interface":"front","average":{"1min":16.012,"5min":16.016999999999999,"15min":19.134},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":7.2080000000000002},"max":{"1min":35.750999999999998,"5min":35.750999999999998,"15min":155.94},"last":15.651999999999999}]},{"osd":2,"last update":"Fri Dec 8 00:18:07 2023","interfaces":[{"interface":"back","average":{"1min":10.789,"5min":11.645,"15min":15.612},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":5.8940000000000001},"max":{"1min":21.337,"5min":21.337,"15min":168.23599999999999},"last":7.3799999999999999},{"interface":"front","average":{"1min":9.5329999999999995,"5min":9.9109999999999996,"15min":15.417999999999999},"min":{"1min":6.3979999999999997,"5min":4.8540000000000001,"15min":4.8540000000000001},"max":{"1min":20.876999999999999,"5min":20.876999999999999,"15min":352.52800000000002},"last":12.73}]},{"osd":3,"last update":"Fri Dec 8 00:18:42 2023","interfaces":[{"interface":"back","average":{"1min":13.205,"5min":11.711,"15min":17.715},"min":{"1min":7.7060000000000004,"5min":6.867,"15min":5.8789999999999996},"max":{"1min":30.916,"5min":30.916,"15min":286.92399999999998},"last":9.2629999999999999},{"interface":"front","average":{"1min":15.866,"5min":15.709,"15min":22.141999999999999},"min":{"1min":6.8010000000000002,"5min":6.8010000000000002,"15min":6.8010000000000002},"max":{"1min":34.753999999999998,"5min":36.771000000000001,"15min":265.44400000000002},"last":17.433}]},{"osd":4,"last update":"Fri Dec 8 00:18:51 2023","interfaces":[{"interface":"back","average":{"1min":12.285,"5min":12.962,"15min":17.344999999999999},"min":{"1min":8.1319999999999997,"5min":6.7389999999999999,"15min":6.7389999999999999},"max":{"1min":18.643000000000001,"5min":33.700000000000003,"15min":139.136},"last":11.41},{"interface":"front","average":{"1min":17.545000000000002,"5min":16.983000000000001,"15min":22.036999999999999},"min":{"1min":13.082000000000001,"5min":9.7469999999999999,"15min":8.8499999999999996},"max":{"1min":33.415999999999997,"5min":37.723999999999997,"15min":156.232},"last":18.257999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:18:07 2023","interfaces":[{"interface":"back","average":{"1min":20.192,"5min":18.701000000000001,"15min":23.984000000000002},"min":{"1min":8.9990000000000006,"5min":8.9990000000000006,"15min":8.9990000000000006},"max":{"1min":70.762,"5min":70.762,"15min":158.316},"last":16.582000000000001},{"interface":"front","average":{"1min":15.691000000000001,"5min":12.763,"15min":17.699999999999999},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.056},"max":{"1min":66.159000000000006,"5min":66.159000000000006,"15min":157.48500000000001},"last":8.6370000000000005}]},{"osd":7,"last update":"Fri Dec 8 00:17:56 2023","interfaces":[{"interface":"back","average":{"1min":13.949999999999999,"5min":14.15,"15min":18.186},"min":{"1min":8.5719999999999992,"5min":8.5719999999999992,"15min":8.4930000000000003},"max":{"1min":30.861000000000001,"5min":30.861000000000001,"15min":152.49600000000001},"last":13.459},{"interface":"front","average":{"1min":18.798999999999999,"5min":18.960000000000001,"15min":23.832999999999998},"min":{"1min":11.927,"5min":11.927,"15min":11.388},"max":{"1min":38.762,"5min":38.762,"15min":299.93099999999998},"last":19.079999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575241,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73316,"kb_used_data":6044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298524,"statfs":{"total":96636764160,"available":96561688576,"internally_reserved":0,"allocated":6189056,"data_stored":3079704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":11.156000000000001,"5min":8.5069999999999997,"15min":14.414999999999999},"min":{"1min":4.4740000000000002,"5min":4.226,"15min":4.226},"max":{"1min":55.738,"5min":55.738,"15min":401.80099999999999},"last":9.4740000000000002},{"interface":"front","average":{"1min":18.251000000000001,"5min":11.83,"15min":15.614000000000001},"min":{"1min":6.1360000000000001,"5min":4.9820000000000002,"15min":4.7690000000000001},"max":{"1min":83.034000000000006,"5min":83.034000000000006,"15min":139.387},"last":7.8490000000000002}]},{"osd":1,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":16.733000000000001,"5min":13.003,"15min":17.408000000000001},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":5.5540000000000003},"max":{"1min":85.837999999999994,"5min":85.837999999999994,"15min":168.71299999999999},"last":8.5120000000000005},{"interface":"front","average":{"1min":24.350000000000001,"5min":17.399999999999999,"15min":23.219999999999999},"min":{"1min":11.083,"5min":10.079000000000001,"15min":10.079000000000001},"max":{"1min":87.222999999999999,"5min":87.222999999999999,"15min":297.87299999999999},"last":18.263000000000002}]},{"osd":2,"last update":"Fri Dec 8 00:18:37 2023","interfaces":[{"interface":"back","average":{"1min":16.844000000000001,"5min":12.420999999999999,"15min":16.123000000000001},"min":{"1min":7.5590000000000002,"5min":5.5460000000000003,"15min":4.6980000000000004},"max":{"1min":66.688999999999993,"5min":66.688999999999993,"15min":212.108},"last":10.834},{"interface":"front","average":{"1min":14.355,"5min":12.999000000000001,"15min":17.460999999999999},"min":{"1min":7.6719999999999997,"5min":6.0209999999999999,"15min":5.7789999999999999},"max":{"1min":62.433999999999997,"5min":88.489000000000004,"15min":402.50200000000001},"last":6.9020000000000001}]},{"osd":3,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.141999999999999,"5min":14.019,"15min":20.75},"min":{"1min":8,"5min":6.6630000000000003,"15min":6.5190000000000001},"max":{"1min":86.459000000000003,"5min":86.459000000000003,"15min":295.20499999999998},"last":14.041},{"interface":"front","average":{"1min":23.57,"5min":16.802,"15min":23.039999999999999},"min":{"1min":7.7939999999999996,"5min":7.7939999999999996,"15min":7.7939999999999996},"max":{"1min":95.340000000000003,"5min":95.340000000000003,"15min":154.19200000000001},"last":12.295999999999999}]},{"osd":4,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":24.349,"5min":18.375,"15min":23.802},"min":{"1min":9.2729999999999997,"5min":9.2729999999999997,"15min":9.0760000000000005},"max":{"1min":84.105999999999995,"5min":84.105999999999995,"15min":337.97199999999998},"last":11.702},{"interface":"front","average":{"1min":20.994,"5min":15.461,"15min":19.864000000000001},"min":{"1min":11.005000000000001,"5min":9.0180000000000007,"15min":6.7350000000000003},"max":{"1min":67.284000000000006,"5min":67.284000000000006,"15min":169.846},"last":13.375999999999999}]},{"osd":6,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":23.216999999999999,"5min":17.192,"15min":19.756},"min":{"1min":9.3260000000000005,"5min":8.4109999999999996,"15min":7.4109999999999996},"max":{"1min":67.876999999999995,"5min":67.876999999999995,"15min":169.93199999999999},"last":15.971},{"interface":"front","average":{"1min":26.831,"5min":19.538,"15min":24.748999999999999},"min":{"1min":14.941000000000001,"5min":10.512,"15min":8.7810000000000006},"max":{"1min":89.180000000000007,"5min":89.180000000000007,"15min":395.5},"last":17.414999999999999}]},{"osd":7,"last update":"Fri Dec 8 00:18:02 2023","interfaces":[{"interface":"back","average":{"1min":25.442,"5min":14.970000000000001,"15min":23.666},"min":{"1min":13.343,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":81.359999999999999,"5min":81.359999999999999,"15min":417.26299999999998},"last":15.346},{"interface":"front","average":{"1min":20.204000000000001,"5min":17.381,"15min":21.201000000000001},"min":{"1min":7.5359999999999996,"5min":7.5359999999999996,"15min":6.5780000000000003},"max":{"1min":84.742000000000004,"5min":84.742000000000004,"15min":352.34100000000001},"last":14.654999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575241,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68764,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303076,"statfs":{"total":96636764160,"available":96566349824,"internally_reserved":0,"allocated":5787648,"data_stored":2681864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 8 00:18:48 2023","interfaces":[{"interface":"back","average":{"1min":8.2219999999999995,"5min":8.9390000000000001,"15min":14.173999999999999},"min":{"1min":5.2030000000000003,"5min":4.46,"15min":4.1429999999999998},"max":{"1min":19.649000000000001,"5min":19.649000000000001,"15min":218.566},"last":5.2030000000000003},{"interface":"front","average":{"1min":9.6010000000000009,"5min":9.8070000000000004,"15min":15.786},"min":{"1min":6.4790000000000001,"5min":5.1520000000000001,"15min":4.8300000000000001},"max":{"1min":17.024000000000001,"5min":19.678999999999998,"15min":299.358},"last":9.3930000000000007}]},{"osd":1,"last update":"Fri Dec 8 00:18:18 2023","interfaces":[{"interface":"back","average":{"1min":14.259,"5min":12.289,"15min":16.882000000000001},"min":{"1min":5.8319999999999999,"5min":5.8319999999999999,"15min":5.8319999999999999},"max":{"1min":58.145000000000003,"5min":58.145000000000003,"15min":291.13200000000001},"last":11.292999999999999},{"interface":"front","average":{"1min":13.249000000000001,"5min":11.358000000000001,"15min":14.342000000000001},"min":{"1min":7.0549999999999997,"5min":5.3490000000000002,"15min":4.9509999999999996},"max":{"1min":55.795000000000002,"5min":55.795000000000002,"15min":119.005},"last":8.6750000000000007}]},{"osd":2,"last update":"Fri Dec 8 00:18:48 2023","interfaces":[{"interface":"back","average":{"1min":12.592000000000001,"5min":12.07,"15min":16.073},"min":{"1min":7.3710000000000004,"5min":6.8819999999999997,"15min":6.1539999999999999},"max":{"1min":21.434000000000001,"5min":86.319000000000003,"15min":300.22399999999999},"last":15.304},{"interface":"front","average":{"1min":15,"5min":14.756,"15min":19.082000000000001},"min":{"1min":8.6120000000000001,"5min":8.6120000000000001,"15min":6.734},"max":{"1min":23.247,"5min":26.940999999999999,"15min":258.58499999999998},"last":16.683}]},{"osd":3,"last update":"Fri Dec 8 00:18:31 2023","interfaces":[{"interface":"back","average":{"1min":13.993,"5min":14.784000000000001,"15min":21.876000000000001},"min":{"1min":10.134,"5min":7.4279999999999999,"15min":6.4219999999999997},"max":{"1min":17.777000000000001,"5min":51.329999999999998,"15min":400.41699999999997},"last":11.962},{"interface":"front","average":{"1min":12.406000000000001,"5min":13.192,"15min":18.846},"min":{"1min":7.9950000000000001,"5min":7.0460000000000003,"15min":7.0460000000000003},"max":{"1min":18.044,"5min":49.639000000000003,"15min":273.839},"last":15.994999999999999}]},{"osd":5,"last update":"Fri Dec 8 00:18:31 2023","interfaces":[{"interface":"back","average":{"1min":15.132999999999999,"5min":15.638,"15min":22.029},"min":{"1min":10.199999999999999,"5min":7.4939999999999998,"15min":7.4939999999999998},"max":{"1min":21.728999999999999,"5min":33.755000000000003,"15min":397.96199999999999},"last":13.294},{"interface":"front","average":{"1min":14.581,"5min":14.759,"15min":20.292999999999999},"min":{"1min":8.6620000000000008,"5min":7.2839999999999998,"15min":6.9900000000000002},"max":{"1min":19.245000000000001,"5min":59.944000000000003,"15min":291.952},"last":17.425000000000001}]},{"osd":6,"last update":"Fri Dec 8 00:18:48 2023","interfaces":[{"interface":"back","average":{"1min":15.667999999999999,"5min":15.446,"15min":19.260000000000002},"min":{"1min":9.0449999999999999,"5min":7.8940000000000001,"15min":6.1399999999999997},"max":{"1min":22.385999999999999,"5min":50.722999999999999,"15min":296.37},"last":19.27},{"interface":"front","average":{"1min":14.747999999999999,"5min":17.068999999999999,"15min":20.056999999999999},"min":{"1min":9.4169999999999998,"5min":9.3499999999999996,"15min":7.8860000000000001},"max":{"1min":22.306999999999999,"5min":111.55,"15min":232.64099999999999},"last":10.494}]},{"osd":7,"last update":"Fri Dec 8 00:18:18 2023","interfaces":[{"interface":"back","average":{"1min":17.148,"5min":17.062999999999999,"15min":22.404},"min":{"1min":10.853,"5min":9.6539999999999999,"15min":7.8659999999999997},"max":{"1min":25.847999999999999,"5min":25.847999999999999,"15min":235.828},"last":18.619},{"interface":"front","average":{"1min":16.129999999999999,"5min":16.532,"15min":22.901},"min":{"1min":10.551,"5min":9.6530000000000005,"15min":9.0869999999999997},"max":{"1min":27.341000000000001,"5min":27.341000000000001,"15min":298.74599999999998},"last":12.664999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T00:18:54.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-08T00:18:54.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.781+0000 7f592a9d5640 10 monclient: get_monmap_and_config 2023-12-08T00:18:54.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.781+0000 7f592a9d5640 10 monclient: build_initial_monmap 2023-12-08T00:18:54.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.781+0000 7f592a9d5640 10 monclient: monmap: 2023-12-08T00:18:54.783 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:54.783 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.783 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-08T00:18:54.782032+0000 2023-12-08T00:18:54.783 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-08T00:18:54.782032+0000 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.129:6789/0 mon.noname-a-legacy 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.184:6789/0 mon.noname-c-legacy 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-b 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr:3: v2:172.21.15.129:3300/0 mon.noname-a 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr:4: v2:172.21.15.184:3300/0 mon.noname-c 2023-12-08T00:18:54.784 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.788 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.782+0000 7f592a9d5640 1 Processor -- start 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.782+0000 7f592a9d5640 1 -- start start 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.782+0000 7f592a9d5640 10 monclient: init 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f592406cdb0 addr v1:172.21.15.129:6789/0 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f592406c1c0 0x7f5924071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient(hunting): picked mon.noname-a con 0x7f592406c1c0 addr v2:172.21.15.129:3300/0 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 0x7f5924071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.789 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient(hunting): picked mon.noname-b con 0x7f592413f510 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592406d4b0 con 0x7f592406cdb0 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592406d190 con 0x7f592406c1c0 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59240721b0 con 0x7f592413f510 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.783+0000 7f592a9d5640 10 monclient(hunting): authenticate will time out at 3334.970309s 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 0x7f5924071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.790 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f592406c1c0 0x7f5924071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f592406c1c0 0x7f5924071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50140/0 (socket says 172.21.15.129:50140) 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 1 -- 172.21.15.129:0/255094178 learned_addr learned my addr 172.21.15.129:0/255094178 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 10 monclient(hunting): get_auth_request con 0x7f592406c1c0 auth_method 0 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 10 monclient(hunting): get_auth_request con 0x7f592413f510 auth_method 0 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f59237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 1 -- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 msgr2=0x7f5924071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-08T00:18:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 1 -- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 msgr2=0x7f5924071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.784+0000 7f5928f72640 1 --2- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 0x7f5924071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.785+0000 7f5928f72640 1 --2- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 0x7f5924071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 10 monclient(hunting): handle_auth_done global_id 15822 payload 274 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 1 -- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 msgr2=0x7f5924071c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 1 --2- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 0x7f5924071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 1 -- 172.21.15.129:0/255094178 >> v1:172.21.15.129:6789/0 conn(0x7f592406cdb0 legacy=0x7f592413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 10 monclient: _finish_hunting 0 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 1 monclient: found mon.noname-a 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.786+0000 7f59237fe640 1 -- 172.21.15.129:0/255094178 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924071030 con 0x7f592406c1c0 2023-12-08T00:18:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.788+0000 7f59237fe640 1 --2- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f5924071730 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7f5914009870 tx=0x7f591402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f2a0838e85df server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 1 -- 172.21.15.129:0/255094178 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914031640 con 0x7f592406c1c0 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: dump: 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: _finish_auth 0 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: _check_auth_tickets 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 1 -- 172.21.15.129:0/255094178 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59140317a0 con 0x7f592406c1c0 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 1 -- 172.21.15.129:0/255094178 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914031b30 con 0x7f592406c1c0 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f5922ffd640 10 monclient: dump: 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f592a9d5640 5 monclient: authenticate success, global_id 15822 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f592a9d5640 10 monclient: get_monmap_and_config success 2023-12-08T00:18:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.792+0000 7f592a9d5640 4 set_mon_vals no callback set 2023-12-08T00:18:54.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 10 monclient: shutdown 2023-12-08T00:18:54.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:54.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 -- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 msgr2=0x7f5924071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 --2- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f5924071730 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7f5914009870 tx=0x7f591402b8f0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.799 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 -- 172.21.15.129:0/255094178 shutdown_connections 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 --2- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592413f510 0x7f5924071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 --2- 172.21.15.129:0/255094178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f5924071730 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 -- 172.21.15.129:0/255094178 >> 172.21.15.129:0/255094178 conn(0x7f59240b5b90 msgr2=0x7f59240b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 -- 172.21.15.129:0/255094178 shutdown_connections 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.793+0000 7f592a9d5640 1 -- 172.21.15.129:0/255094178 wait complete. 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 10 monclient: build_initial_monmap 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 10 monclient: monmap: 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.noname-a 2023-12-08T00:18:54.800 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.noname-b 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.noname-c 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 1 librados: starting msgr at 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 1 librados: starting objecter 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 1 Processor -- start 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 1 -- start start 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 1 librados: setting wanted keys 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 1 librados: calling monclient init 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.794+0000 7f592a9d5640 10 monclient: init 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient: _reopen_session rank -1 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f59240aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): picked mon.noname-a con 0x7f592406c1c0 addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592406cdb0 0x7f59241da700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.801 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): picked mon.noname-c con 0x7f592406cdb0 addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f592413f510 0x7f5924135a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): picked mon.noname-b con 0x7f592413f510 addr [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924140d80 con 0x7f592406c1c0 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592413ff90 con 0x7f592413f510 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): start opening mon connection 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5924140ff0 con 0x7f592406cdb0 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): _renew_subs 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f592a9d5640 10 monclient(hunting): authenticate will time out at 3334.982532s 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.795+0000 7f5923fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f59240aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f59237fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592406cdb0 0x7f59241da700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f59240aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50142/0 (socket says 172.21.15.129:50142) 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 1 -- 172.21.15.129:0/2645943920 learned_addr learned my addr 172.21.15.129:0/2645943920 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-12-08T00:18:54.802 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 10 monclient(hunting): get_auth_request con 0x7f592406c1c0 auth_method 0 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5928f72640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f592413f510 0x7f5924135a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f59237fe640 10 monclient(hunting): get_auth_request con 0x7f592406cdb0 auth_method 0 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f59237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f59237fe640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f59237fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5928f72640 10 monclient(hunting): get_auth_request con 0x7f592413f510 auth_method 0 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5928f72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5928f72640 10 monclient(hunting): _init_auth method 2 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5928f72640 10 monclient(hunting): _init_auth creating new auth 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f5923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.796+0000 7f59237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.797+0000 7f59237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.797+0000 7f59237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.797+0000 7f5928f72640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-08T00:18:54.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.797+0000 7f5928f72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-08T00:18:54.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.797+0000 7f5928f72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 10 monclient(hunting): handle_auth_done global_id 15825 payload 824 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 1 -- 172.21.15.129:0/2645943920 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592406cdb0 msgr2=0x7f59241da700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592406cdb0 0x7f59241da700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 1 -- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f592413f510 msgr2=0x7f5924135a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f592413f510 0x7f5924135a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 10 monclient: _finish_hunting 0 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 1 monclient: found mon.noname-a 2023-12-08T00:18:54.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-08T00:18:54.806 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.806 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.799+0000 7f5923fff640 1 -- 172.21.15.129:0/2645943920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59140094d0 con 0x7f592406c1c0 2023-12-08T00:18:54.806 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.801+0000 7f5923fff640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f59240aef80 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7f590c00c410 tx=0x7f590c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:54.806 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.801+0000 7f5920ff9640 10 client.?.objecter ms_handle_connect 0x7f592406c1c0 2023-12-08T00:18:54.806 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.801+0000 7f5920ff9640 10 client.?.objecter resend_mon_ops 2023-12-08T00:18:54.807 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c0035f0 con 0x7f592406c1c0 2023-12-08T00:18:54.807 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: dump: 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: _finish_auth 0 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: _check_auth_tickets 2023-12-08T00:18:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c003750 con 0x7f592406c1c0 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 5 monclient: authenticate success, global_id 15825 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 10 monclient: _renew_subs 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c003b40 con 0x7f592406c1c0 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5924137790 con 0x7f592406c1c0 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f59227fc640 4 set_mon_vals no callback set 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f5920ff9640 10 monclient: dump: 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:fsid a6eef5de-0da3-4292-875d-ef3a20551b59 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:created 2023-12-07T23:44:30.440011+0000 2023-12-08T00:18:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 18 (reef) 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 10 client.15825.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 10 monclient: _renew_subs 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5924142b50 con 0x7f592406c1c0 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.805+0000 7f592a9d5640 1 librados: init done 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.806+0000 7f592a9d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.806+0000 7f592a9d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.806+0000 7f592a9d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.806+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592413fe00 con 0x7f592406c1c0 2023-12-08T00:18:54.826 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.823+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f590c003ca0 con 0x7f592406c1c0 2023-12-08T00:18:54.826 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.824+0000 7f5920ff9640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f590406c870 0x7f590406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-08T00:18:54.826 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.824+0000 7f59237fe640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f590406c870 0x7f590406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-08T00:18:54.826 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.824+0000 7f59237fe640 10 monclient: get_auth_request con 0x7f590406c870 auth_method 0 2023-12-08T00:18:54.826 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.824+0000 7f5920ff9640 10 client.15825.objecter ms_handle_connect 0x7f590406c870 2023-12-08T00:18:54.826 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.824+0000 7f59237fe640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f590406c870 0x7f590406ed30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5914036000 tx=0x7f591403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T00:18:54.829 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.827+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1049..1049 src has 999..1049) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f590c057f10 con 0x7f592406c1c0 2023-12-08T00:18:54.829 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.827+0000 7f5920ff9640 10 client.15825.objecter ms_dispatch 0x7f5924000b90 osd_map(1049..1049 src has 999..1049) v4 2023-12-08T00:18:54.829 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.827+0000 7f5920ff9640 3 client.15825.objecter handle_osd_map got epochs [1049,1049] > 0 2023-12-08T00:18:54.829 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.827+0000 7f5920ff9640 3 client.15825.objecter handle_osd_map decoding full epoch 1049 2023-12-08T00:18:54.829 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:54.827+0000 7f5920ff9640 20 client.15825.objecter dump_active .. 0 homeless 2023-12-08T00:18:55.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.061+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f590c054250 con 0x7f592406c1c0 2023-12-08T00:18:55.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.061+0000 7f5920ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-08T00:18:55.062 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.061+0000 7f5920ff9640 10 monclient: _finish_command 1 = system:0 2023-12-08T00:18:55.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.233+0000 7f592a9d5640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-08T00:18:55.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.233+0000 7f592a9d5640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-08T00:18:55.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.233+0000 7f592a9d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.129:3300/0 2023-12-08T00:18:55.235 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.233+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f5924138370 con 0x7f592406c1c0 2023-12-08T00:18:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.340+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f590c0170c0 con 0x7f592406c1c0 2023-12-08T00:18:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.340+0000 7f5920ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-08T00:18:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.340+0000 7f5920ff9640 10 monclient: _finish_command 2 = system:0 2023-12-08T00:18:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.340+0000 7f5920ff9640 1 -- 172.21.15.129:0/2645943920 <== mon.0 v2:172.21.15.129:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f590c05e0d0 con 0x7f592406c1c0 2023-12-08T00:18:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.340+0000 7f5920ff9640 10 monclient: handle_config config(1 keys) v1 2023-12-08T00:18:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.340+0000 7f59227fc640 4 set_mon_vals no callback set 2023-12-08T00:18:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.340+0000 7f59227fc640 10 set_mon_vals mon_health_to_clog = false 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 10 librados: watch_flush enter 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 10 librados: watch_flush exit 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 20 client.15825.objecter shutdown clearing up homeless session... 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 10 client.15825.objecter successfully canceled tick 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f590406c870 msgr2=0x7f590406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f590406c870 0x7f590406ed30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5914036000 tx=0x7f591403c040 comp rx=0 tx=0).stop 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 10 monclient: shutdown 2023-12-08T00:18:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 msgr2=0x7f59240aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f59240aef80 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7f590c00c410 tx=0x7f590c00c8d0 comp rx=0 tx=0).stop 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 shutdown_connections 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:6800/2561365197,v1:172.21.15.184:6801/2561365197] conn(0x7f590406c870 0x7f590406ed30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f592413f510 0x7f5924135a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f592406cdb0 0x7f59241da700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 --2- 172.21.15.129:0/2645943920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f592406c1c0 0x7f59240aef80 unknown :-1 s=CLOSED pgs=3285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 >> 172.21.15.129:0/2645943920 conn(0x7f59240b5b90 msgr2=0x7f59240b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 shutdown_connections 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 -- 172.21.15.129:0/2645943920 wait complete. 2023-12-08T00:18:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2023-12-08T00:18:55.343+0000 7f592a9d5640 1 librados: shutdown 2023-12-08T00:18:55.356 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-08T00:18:55.357 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-08T00:18:55.357 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-08T00:18:55.357 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:18:55.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-08T00:18:55.360+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 87028) UID: 0 2023-12-08T00:18:55.368 INFO:tasks.ceph.osd.0.smithi129.stderr:2023-12-08T00:18:55.361+0000 b47e640 -1 osd.0 1049 *** Got signal Terminated *** 2023-12-08T00:18:56.079 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:52.553 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.125 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:52.570 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.126 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:52.600 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.127 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:52.573 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.142 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:52.588 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.144 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:52.618 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.180 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:52.654 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.181 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:52.655 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:18:56.194 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:52.668 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.094 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:57.568 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.141 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:57.614 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.141 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:57.587 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.144 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:57.590 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.158 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:33:57.604 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.160 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:33:57.634 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.209 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:57.682 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.212 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:57.686 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.224 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:33:57.697 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:01.459 INFO:tasks.ceph.osd.0:Stopped 2023-12-08T00:19:01.459 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-08T00:19:01.459 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:19:01.470 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-08T00:19:01.461+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 87032) UID: 0 2023-12-08T00:19:01.471 INFO:tasks.ceph.osd.1.smithi129.stderr:2023-12-08T00:19:01.462+0000 b47e640 -1 osd.1 1051 *** Got signal Terminated *** 2023-12-08T00:19:16.177 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:12.650 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.179 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:12.653 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.181 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:12.626 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.185 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:12.630 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.197 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:12.671 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.209 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:12.653 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.227 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:12.700 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.231 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:12.704 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:16.247 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:12.720 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.196 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:17.669 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.198 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:17.671 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.213 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:17.686 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.244 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:17.717 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.246 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:17.720 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.247 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:17.693 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.250 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:17.696 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.258 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:17.732 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:21.262 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:17.708 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.251 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:32.725 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.255 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:32.729 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.272 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:32.746 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.304 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:32.777 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.308 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:32.781 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.310 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:32.755 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.311 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:32.757 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.323 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:32.797 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:36.327 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:32.773 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.249 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:37.722 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.253 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:37.727 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.267 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:37.740 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.300 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:37.745 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.301 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:37.747 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.312 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:37.758 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.323 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:37.796 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.324 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:37.798 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:19:41.334 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:37.808 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.298 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:57.772 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.301 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:57.775 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.314 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:34:57.788 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.342 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:57.788 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.345 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:57.791 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.349 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:57.822 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.350 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:57.824 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.361 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:34:57.807 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:01.366 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:34:57.839 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:02.474 INFO:tasks.ceph.osd.1:Stopped 2023-12-08T00:20:02.475 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-08T00:20:02.475 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:20:02.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-08T00:20:02.478+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 87043) UID: 0 2023-12-08T00:20:02.492 INFO:tasks.ceph.osd.2.smithi129.stderr:2023-12-08T00:20:02.480+0000 b47e640 -1 osd.2 1053 *** Got signal Terminated *** 2023-12-08T00:20:08.576 INFO:tasks.ceph.osd.2:Stopped 2023-12-08T00:20:08.577 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-08T00:20:08.577 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:20:08.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-08T00:20:08.579+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 87047) UID: 0 2023-12-08T00:20:08.588 INFO:tasks.ceph.osd.3.smithi129.stderr:2023-12-08T00:20:08.580+0000 b47e640 -1 osd.3 1056 *** Got signal Terminated *** 2023-12-08T00:20:12.931 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:09.405 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.937 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:09.410 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.942 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:09.388 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.943 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:09.416 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.944 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:09.390 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.945 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:09.419 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.961 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:09.435 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.967 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:09.413 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:12.981 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:09.455 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.402 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:17.876 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.406 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:17.880 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.425 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:17.899 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.450 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:17.923 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.455 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:17.929 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.457 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:17.903 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.459 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:17.904 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.472 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:17.917 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:21.472 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:17.945 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.489 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:32.963 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.494 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:32.968 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.514 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:32.988 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.540 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:32.986 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.543 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:33.016 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.545 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:32.990 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.546 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:33.019 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.562 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:33.035 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:36.562 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:33.007 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.505 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:37.979 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.509 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:37.983 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.522 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:37.996 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.550 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:38.024 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.553 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:38.026 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.557 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:38.003 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.562 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:38.007 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.564 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:38.037 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:41.572 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:38.017 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.539 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:53.012 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.543 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:53.017 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.560 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:53.034 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.587 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:53.033 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.591 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:53.065 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.594 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:53.040 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.598 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:53.072 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.608 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:53.053 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:20:56.617 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:53.090 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.561 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:58.035 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.563 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:58.037 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.573 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:35:58.047 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.615 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:58.060 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.615 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:58.088 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.616 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:58.090 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.617 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:58.063 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.626 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:35:58.099 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:01.627 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:35:58.073 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:03.489 INFO:tasks.ceph.osd.3:Stopped 2023-12-08T00:21:03.489 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-08T00:21:03.489 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:21:03.504 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-08T00:21:03.492+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 87192) UID: 0 2023-12-08T00:21:03.505 INFO:tasks.ceph.osd.4.smithi184.stderr:2023-12-08T00:21:03.493+0000 b47e640 -1 osd.4 1059 *** Got signal Terminated *** 2023-12-08T00:21:09.591 INFO:tasks.ceph.osd.4:Stopped 2023-12-08T00:21:09.591 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-08T00:21:09.591 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:21:09.604 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-08T00:21:09.594+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 87195) UID: 0 2023-12-08T00:21:09.607 INFO:tasks.ceph.osd.5.smithi184.stderr:2023-12-08T00:21:09.595+0000 b47e640 -1 osd.5 1061 *** Got signal Terminated *** 2023-12-08T00:21:15.693 INFO:tasks.ceph.osd.5:Stopped 2023-12-08T00:21:15.693 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-08T00:21:15.693 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:21:15.706 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-08T00:21:15.696+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 87197) UID: 0 2023-12-08T00:21:15.706 INFO:tasks.ceph.osd.6.smithi184.stderr:2023-12-08T00:21:15.697+0000 b47e640 -1 osd.6 1063 *** Got signal Terminated *** 2023-12-08T00:21:16.604 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:13.077 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.606 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:13.080 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.637 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:13.111 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.651 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:13.096 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.651 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:13.124 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.652 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:13.098 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.655 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:13.128 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.673 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:13.119 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:16.676 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:13.149 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.064 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:13.509 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.068 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:13.542 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.069 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:13.543 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.079 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:13.553 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.102 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:13.576 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.103 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:13.577 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.111 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:13.585 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.114 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:13.559 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:17.122 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:13.568 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.731 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:33.205 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.778 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:33.251 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.795 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:33.268 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.825 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:33.298 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.825 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:33.271 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.826 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:33.272 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.826 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:33.300 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.842 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:33.287 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:36.842 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:33.316 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:50.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-12-08T00:21:52.302 INFO:tasks.ceph.osd.6:Stopped 2023-12-08T00:21:52.303 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-08T00:21:52.303 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:21:52.316 INFO:tasks.ceph.osd.7.smithi184.stderr:2023-12-08T00:21:52.305+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 87199) UID: 0 2023-12-08T00:21:52.316 INFO:tasks.ceph.osd.7.smithi184.stderr:2023-12-08T00:21:52.307+0000 b47e640 -1 osd.7 1063 *** Got signal Terminated *** 2023-12-08T00:21:56.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-12-08T00:21:56.726 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:53.200 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.730 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:53.204 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.757 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:53.231 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.782 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:53.227 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.783 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:53.229 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.786 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:53.259 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.790 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:53.264 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.799 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:53.245 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:56.808 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:53.281 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.286 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:53.760 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.287 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:53.761 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.296 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:36:53.770 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.306 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:53.751 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.306 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:53.752 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.315 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:36:53.761 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.320 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:53.793 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.323 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:53.796 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:21:57.333 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:36:53.806 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.786 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:13.260 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.788 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:13.262 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.806 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:13.279 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.835 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:13.308 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.840 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:13.285 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.840 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:13.314 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.841 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:13.287 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.858 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:13.303 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:16.858 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:13.331 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.355 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:13.828 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.356 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:13.830 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.366 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:13.840 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.369 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:13.815 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.370 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:13.816 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.378 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:13.823 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.393 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:13.867 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.395 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:13.868 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:17.403 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:13.876 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.793 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:33.266 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.840 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:33.314 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.860 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:33.334 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.885 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:33.330 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.887 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:33.333 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.892 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:33.365 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.895 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:33.369 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.905 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:33.351 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:36.914 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:33.387 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.398 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:33.872 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.399 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:33.873 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.407 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:33.880 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.412 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:33.857 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.412 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:33.858 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.420 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:33.866 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.438 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:33.912 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.439 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:33.913 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:37.448 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:33.921 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.898 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:53.371 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.899 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:53.372 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.916 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:53.390 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.942 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:53.387 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.946 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:53.419 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.946 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:53.392 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.948 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:53.421 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.964 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:53.410 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:56.966 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:53.439 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.455 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:53.929 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.456 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:53.930 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.463 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:37:53.937 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.476 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:53.922 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.477 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:53.923 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.485 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:37:53.931 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.493 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:53.966 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.494 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:53.967 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:22:57.502 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:37:53.975 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:21.963 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:18.436 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:21.969 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:18.443 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:21.992 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:18.464 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.017 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:18.462 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.020 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:18.493 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.021 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:18.467 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.023 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:18.497 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.043 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:18.488 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.043 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:18.517 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.627 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:19.071 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.627 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:19.073 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.635 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:19.081 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.662 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:19.135 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.663 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:19.136 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.665 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:19.139 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.676 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:19.140 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.678 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:19.145 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:22.678 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:19.149 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.625 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:23.099 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.628 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:23.102 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.633 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:23.079 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.635 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:23.108 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.635 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:23.081 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.638 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:23.111 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.647 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:23.121 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.652 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:23.098 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:26.656 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:23.129 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.128 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:38.601 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.131 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:38.604 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.150 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:38.623 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.176 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:38.621 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.180 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:38.653 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.181 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:38.626 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.181 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:38.655 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.196 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:38.641 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.198 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:38.672 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.687 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:39.133 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.688 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:39.134 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.697 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:39.143 86583== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.723 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:39.197 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.724 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:39.198 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.734 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:39.208 86737== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.737 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:39.211 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.738 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:39.212 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:42.747 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:39.221 86742== Warning: unimplemented fcntl command: 1036 2023-12-08T00:23:48.232 INFO:tasks.ceph.osd.7:Stopped 2023-12-08T00:23:48.232 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-08T00:23:48.233 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-08T00:23:48.233 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:23:48.259 INFO:tasks.ceph.mgr.y:Stopped 2023-12-08T00:23:48.259 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-08T00:23:48.259 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:23:48.260 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-08T00:23:48.258+0000 7f57d1cfb640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 86581) UID: 0 2023-12-08T00:23:48.260 INFO:tasks.ceph.mgr.x.smithi184.stderr:2023-12-08T00:23:48.258+0000 7f57d1cfb640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-08T00:23:48.286 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-08T00:23:48.286 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-12-08T00:23:48.286 INFO:tasks.ceph.mgr.x:Stopped 2023-12-08T00:23:48.286 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-08T00:23:48.287 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-08T00:23:48.287 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:23:48.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2023-12-08T00:23:48.301+0000 9f74640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86708) UID: 0 2023-12-08T00:23:48.314 INFO:tasks.ceph.mon.a.smithi129.stderr:2023-12-08T00:23:48.303+0000 9f74640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-08T00:23:49.042 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:45.516 86737== 2023-12-08T00:23:49.042 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:38:45.516 86737== Exit program on first error (--exit-on-first-error=yes) 2023-12-08T00:23:49.064 INFO:tasks.ceph.mon.a.smithi129.stderr:daemon-helper: command failed with exit status 42 2023-12-08T00:23:53.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-08T00:23:53.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-08T00:23:53.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-12-08T00:23:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-08T00:23:54.389 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-08T00:23:54.390 INFO:tasks.ceph.mon.a:Stopped 2023-12-08T00:23:54.391 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-08T00:23:54.391 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:23:54.406 INFO:tasks.ceph.mon.c.smithi129.stderr:2023-12-08T00:23:54.393+0000 9f74640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86710) UID: 0 2023-12-08T00:23:54.410 INFO:tasks.ceph.mon.c.smithi129.stderr:2023-12-08T00:23:54.394+0000 9f74640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-08T00:23:55.147 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:51.620 86742== 2023-12-08T00:23:55.148 INFO:tasks.ceph.mon.c.smithi129.stderr:==00:00:38:51.621 86742== Exit program on first error (--exit-on-first-error=yes) 2023-12-08T00:23:55.169 INFO:tasks.ceph.mon.c.smithi129.stderr:daemon-helper: command failed with exit status 42 2023-12-08T00:23:58.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-08T00:23:58.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-08T00:24:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-08T00:24:00.493 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-08T00:24:00.493 INFO:tasks.ceph.mon.c:Stopped 2023-12-08T00:24:00.494 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-08T00:24:00.494 INFO:teuthology.orchestra.run:waiting for 300 2023-12-08T00:24:00.508 INFO:tasks.ceph.mon.b.smithi184.stderr:2023-12-08T00:24:00.496+0000 9f74640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86579) UID: 0 2023-12-08T00:24:00.509 INFO:tasks.ceph.mon.b.smithi184.stderr:2023-12-08T00:24:00.497+0000 9f74640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-12-08T00:24:01.302 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:57.748 86583== 2023-12-08T00:24:01.303 INFO:tasks.ceph.mon.b.smithi184.stderr:==00:00:38:57.748 86583== Exit program on first error (--exit-on-first-error=yes) 2023-12-08T00:24:01.322 INFO:tasks.ceph.mon.b.smithi184.stderr:daemon-helper: command failed with exit status 42 2023-12-08T00:24:03.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-08T00:24:03.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-08T00:24:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-08T00:24:06.596 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi184 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-08T00:24:06.597 INFO:tasks.ceph.mon.b:Stopped 2023-12-08T00:24:06.597 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-08T00:24:06.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-12-08T00:24:06.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-08T00:24:06.636 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-08T00:24:06.759 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-08T00:24:06.759 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-08T00:24:06.823 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-08T00:24:06.823 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-08T00:24:06.883 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi129.front.sepia.ceph.com 2023-12-08T00:24:06.883 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-08T00:24:06.950 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-08T00:24:06.950 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-08T00:24:07.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-08T00:24:07.053 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-08T00:24:07.134 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-08T00:24:07.134 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-08T00:24:07.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi184.front.sepia.ceph.com 2023-12-08T00:24:07.191 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-08T00:24:07.249 INFO:tasks.ceph:Archiving mon data... 2023-12-08T00:24:07.251 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/data/mon.a.tgz 2023-12-08T00:24:07.251 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2023-12-08T00:24:07.267 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.7NHDuJG1kS 2023-12-08T00:24:07.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7NHDuJG1kS 2023-12-08T00:24:07.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.7NHDuJG1kS 2023-12-08T00:24:07.501 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.7NHDuJG1kS is 423KB 2023-12-08T00:24:07.599 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.7NHDuJG1kS 2023-12-08T00:24:07.613 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/data/mon.c.tgz 2023-12-08T00:24:07.614 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2023-12-08T00:24:07.667 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.hPA3lFMPOw 2023-12-08T00:24:07.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hPA3lFMPOw 2023-12-08T00:24:07.802 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.hPA3lFMPOw 2023-12-08T00:24:07.887 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.hPA3lFMPOw is 422KB 2023-12-08T00:24:07.984 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.hPA3lFMPOw 2023-12-08T00:24:07.999 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/data/mon.b.tgz 2023-12-08T00:24:07.999 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2023-12-08T00:24:08.017 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.c1TK2VNk4m 2023-12-08T00:24:08.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.c1TK2VNk4m 2023-12-08T00:24:08.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.c1TK2VNk4m 2023-12-08T00:24:08.277 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.c1TK2VNk4m is 422KB 2023-12-08T00:24:08.347 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.c1TK2VNk4m 2023-12-08T00:24:08.362 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-08T00:24:08.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-08T00:24:08.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-08T00:24:08.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-08T00:24:08.749 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-08T00:24:08.990 INFO:tasks.ceph:Archiving crash dumps... 2023-12-08T00:24:08.992 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/remote/smithi129/crash 2023-12-08T00:24:08.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-08T00:24:09.019 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/remote/smithi184/crash 2023-12-08T00:24:09.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-08T00:24:09.051 INFO:tasks.ceph:Compressing logs... 2023-12-08T00:24:09.051 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-08T00:24:09.066 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-08T00:24:09.091 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92858.log 2023-12-08T00:24:09.091 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160048.log 2023-12-08T00:24:09.092 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92448.log 2023-12-08T00:24:09.092 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160048.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-08T00:24:09.092 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90857.log 2023-12-08T00:24:09.092 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.160048.log.gz 2023-12-08T00:24:09.092 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160552.log 2023-12-08T00:24:09.092 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.90857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160384.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86736.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:gzip/var/log/ceph/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.87027.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.86736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86896.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.87027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86856.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.86736.log.gz 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.86896.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87027.log.gz 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.86856.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86896.log.gz 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-12-08T00:24:09.139 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/valgrind/osd.7.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86856.log.gz 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/valgrind/osd.5.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr: 63.8% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/valgrind/osd.4.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86816.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.86816.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86940.log 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86816.log.gz 2023-12-08T00:24:09.140 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86987.log 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.86940.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.86987.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86940.log.gz 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mgr.x.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86987.log.gz 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86776.log 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mon.b.log: 92.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.86776.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86776.log.gz 2023-12-08T00:24:09.141 INFO:teuthology.orchestra.run.smithi184.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159922.log 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90857.log.gz 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92117.log 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160552.log.gz 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.159922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91335.log 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160384.log.gz 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.92117.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86750.log 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91335.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159922.log.gz 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108869.log 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92117.log.gz 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86750.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91335.log.gz 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159964.log 2023-12-08T00:24:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93345.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108869.log: /var/log/ceph/ceph-client.admin.159964.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93345.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.86750.log.gz 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.108869.log.gz 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159964.log.gz 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/valgrind/mon.c.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/valgrind/osd.0.log: -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr: 64.0%/var/log/ceph/valgrind/osd.1.log: -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92940.log 2023-12-08T00:24:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/valgrind/mon.a.log: /var/log/ceph/valgrind/osd.3.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90721.log 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160132.log 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160510.log 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160132.log: gzip -5 88.2% --verbose -- /var/log/ceph/ceph-client.admin.91947.log 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90721.log.gz 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0%/var/log/ceph/ceph-client.admin.160510.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.160132.log.gz 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91839.log 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91947.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92858.log.gz 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91035.log 2023-12-08T00:24:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91839.log: gzip 87.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.91030.log 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91947.log.gz 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93345.log.gz 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91035.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160510.log.gz 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110779.log 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91839.log.gz 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93569.log 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91035.log.gz 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.110779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92075.log 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91030.log.gz 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159838.log 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5/var/log/ceph/ceph-client.admin.92075.log: --verbose -- /var/log/ceph/ceph-client.admin.160594.log 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.159838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160300.log 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92075.log.gz 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.159838.log.gz/var/log/ceph/ceph-client.admin.160594.log: 91.2% 2023-12-08T00:24:09.156 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92940.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93261.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117645.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.110779.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.160594.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93261.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160300.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.93310.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.117645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92027.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91376.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93351.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92027.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91376.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.93310.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92500.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90767.log 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91376.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: 91.8%/var/log/ceph/ceph-client.admin.92500.log: -- replaced with /var/log/ceph/ceph-client.admin.93261.log.gz 2023-12-08T00:24:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91907.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.117645.log.gz 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160090.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92159.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90767.log.gz 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160090.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91907.log.gz 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160802.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93032.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160802.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.160090.log.gz 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92159.log.gz 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90815.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160802.log.gz 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93032.log: /var/log/ceph/ceph-client.admin.90815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92765.log 2023-12-08T00:24:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90815.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87079.log 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92765.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.93569.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose 91.6% -- replaced with /var/log/ceph/ceph-client.admin.93351.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.92659.log 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110778.log 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92659.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87079.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137706.log 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.110778.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93032.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93474.log 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.137706.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92500.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87052.log 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93474.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.110778.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88912.log 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87052.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.137706.log.gz 2023-12-08T00:24:09.159 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87052.log.gz 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92947.log 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90681.log 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88912.log.gz 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91036.log 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90681.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92765.log.gz 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90681.log.gz 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.117647.log 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91595.log 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.117647.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91036.log.gz 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91571.log 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91595.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91595.log.gz 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123257.log 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91571.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91571.log.gz 95.4% 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-08T00:24:09.160 INFO:teuthology.orchestra.run.smithi129.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.117647.log.gz 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.92448.log.gz 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123255.log 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.123257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160174.log 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.123255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90213.log 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160174.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90213.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.160174.log.gz 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108868.log 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90213.log.gz 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.1.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.123257.log.gz 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91039.log 2023-12-08T00:24:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91325.log 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108868.log: /var/log/ceph/ceph-client.admin.91039.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.123255.log.gz 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91039.log.gz 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92867.log 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137705.log 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92867.log: 88.6% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92947.log.gz 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91325.log.gz 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91333.log 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.137705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128259.log 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91333.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.108868.log.gz 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91333.log.gz 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91691.log 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.128259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92845.log 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91691.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.137705.log.gz 2023-12-08T00:24:09.162 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91691.log.gz 2023-12-08T00:24:09.163 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128264.log 2023-12-08T00:24:09.163 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160468.log 2023-12-08T00:24:09.163 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.128264.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92867.log.gz 2023-12-08T00:24:09.163 INFO:teuthology.orchestra.run.smithi129.stderr:gzip 90.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.160426.log 2023-12-08T00:24:09.163 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.128259.log.gz 2023-12-08T00:24:09.163 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90617.log 2023-12-08T00:24:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160426.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160468.log.gz 2023-12-08T00:24:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92692.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.160426.log.gz 2023-12-08T00:24:09.165 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:24:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93600.log 2023-12-08T00:24:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92692.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90617.log.gz 2023-12-08T00:24:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91026.log 2023-12-08T00:24:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93600.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.128264.log.gz 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90570.log 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91026.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91026.log.gz 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160342.log 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90570.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90570.log.gz 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160342.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.160342.log.gz 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91024.log 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92845.log.gz 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93595.log 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91024.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91024.log.gz 2023-12-08T00:24:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91690.log 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91751.log 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91690.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91690.log.gz 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.93595.log.gz 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160258.log 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91751.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91751.log.gz 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87188.log 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160258.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160258.log.gz 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93544.log 2023-12-08T00:24:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87188.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87188.log.gz 2023-12-08T00:24:09.171 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160006.log 2023-12-08T00:24:09.172 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93544.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.93544.log.gz 2023-12-08T00:24:09.172 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128260.log 2023-12-08T00:24:09.172 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160006.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160006.log.gz 2023-12-08T00:24:09.172 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137707.log 2023-12-08T00:24:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.128260.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.128260.log.gz 2023-12-08T00:24:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92549.log 2023-12-08T00:24:09.178 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.137707.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.93474.log.gz 2023-12-08T00:24:09.180 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93426.log 2023-12-08T00:24:09.180 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92549.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.137707.log.gz 2023-12-08T00:24:09.180 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91334.log 2023-12-08T00:24:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93426.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92549.log.gz 2023-12-08T00:24:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93207.log 2023-12-08T00:24:09.182 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91334.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91334.log.gz 2023-12-08T00:24:09.189 INFO:teuthology.orchestra.run.smithi129.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.93600.log.gz 2023-12-08T00:24:09.190 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80899.log 2023-12-08T00:24:09.190 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93347.log 2023-12-08T00:24:09.191 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.tmp-client.admin.80899.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80899.log.gz 2023-12-08T00:24:09.193 INFO:teuthology.orchestra.run.smithi129.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.93207.log.gz 2023-12-08T00:24:09.196 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110780.log 2023-12-08T00:24:09.196 INFO:teuthology.orchestra.run.smithi129.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.93426.log.gz 2023-12-08T00:24:09.196 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93431.log 2023-12-08T00:24:09.196 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-08T00:24:09.196 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.110780.log: /var/log/ceph/ceph-client.admin.93431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160762.log 2023-12-08T00:24:09.196 INFO:teuthology.orchestra.run.smithi129.stderr: 91.8%/var/log/ceph/ceph-mgr.y.log: -- replaced with /var/log/ceph/ceph-client.admin.93347.log.gz 2023-12-08T00:24:09.199 INFO:teuthology.orchestra.run.smithi129.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93431.log.gz 2023-12-08T00:24:09.199 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87992.log 2023-12-08T00:24:09.199 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160762.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.110780.log.gz 2023-12-08T00:24:09.199 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.160762.log.gz 2023-12-08T00:24:09.200 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91789.log 2023-12-08T00:24:09.200 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93565.log 2023-12-08T00:24:09.200 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91789.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87992.log.gz 2023-12-08T00:24:09.200 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91789.log.gz 2023-12-08T00:24:09.200 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92977.log 2023-12-08T00:24:09.201 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91033.log 2023-12-08T00:24:09.204 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93565.log: /var/log/ceph/ceph-client.admin.92977.log: 94.3% -- replaced with /var/log/ceph/ceph-client.admin.92659.log.gz 2023-12-08T00:24:09.205 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92537.log 2023-12-08T00:24:09.206 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91033.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91033.log.gz 2023-12-08T00:24:09.206 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91019.log 2023-12-08T00:24:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92537.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92977.log.gz 2023-12-08T00:24:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123256.log 2023-12-08T00:24:09.210 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91019.log.gz 2023-12-08T00:24:09.210 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93160.log 2023-12-08T00:24:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.123256.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.123256.log.gz 2023-12-08T00:24:09.212 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91330.log 2023-12-08T00:24:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93160.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.93160.log.gz 2023-12-08T00:24:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93202.log 2023-12-08T00:24:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91330.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91330.log.gz 2023-12-08T00:24:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160720.log 2023-12-08T00:24:09.217 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93202.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93202.log.gz 2023-12-08T00:24:09.217 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91967.log 2023-12-08T00:24:09.218 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160720.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.93565.log.gz 2023-12-08T00:24:09.218 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.160720.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93018.log 2023-12-08T00:24:09.218 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:24:09.218 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93083.log 2023-12-08T00:24:09.219 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91967.log: /var/log/ceph/ceph-client.admin.93018.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91967.log.gz 2023-12-08T00:24:09.219 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-08T00:24:09.222 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93083.log: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-08T00:24:09.224 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91373.log 2023-12-08T00:24:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.0.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93018.log.gz 2023-12-08T00:24:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160678.log 2023-12-08T00:24:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91373.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91373.log.gz 2023-12-08T00:24:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159880.log 2023-12-08T00:24:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160678.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.160678.log.gz 2023-12-08T00:24:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93135.log 2023-12-08T00:24:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.159880.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.159880.log.gz 2023-12-08T00:24:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91332.log 2023-12-08T00:24:09.240 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93135.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.93135.log.gz 2023-12-08T00:24:09.240 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160636.log 2023-12-08T00:24:09.241 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91332.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91332.log.gz 2023-12-08T00:24:09.242 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-08T00:24:09.243 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.160636.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160636.log.gz 2023-12-08T00:24:09.243 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93308.log 2023-12-08T00:24:09.256 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.log: 92.7%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-08T00:24:09.261 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93308.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93308.log.gz 2023-12-08T00:24:09.261 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-12-08T00:24:09.262 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92583.log 2023-12-08T00:24:09.269 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.2.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.92537.log.gz 2023-12-08T00:24:09.271 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92760.log 2023-12-08T00:24:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92583.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92583.log.gz 2023-12-08T00:24:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92769.log 2023-12-08T00:24:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92760.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.93083.log.gz 2023-12-08T00:24:09.283 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92760.log.gz 2023-12-08T00:24:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160216.log 2023-12-08T00:24:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128257.log 2023-12-08T00:24:09.285 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92769.log: /var/log/ceph/ceph-client.admin.160216.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.160216.log.gz 2023-12-08T00:24:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93260.log 2023-12-08T00:24:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.128257.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.128257.log.gz 2023-12-08T00:24:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93497.log 2023-12-08T00:24:09.302 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93260.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93260.log.gz 2023-12-08T00:24:09.302 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128256.log 2023-12-08T00:24:09.316 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92969.log 2023-12-08T00:24:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.128256.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.128256.log.gz 2023-12-08T00:24:09.321 INFO:teuthology.orchestra.run.smithi129.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.admin.93497.log.gz 2023-12-08T00:24:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92574.log 2023-12-08T00:24:09.337 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92467.log 2023-12-08T00:24:09.343 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92574.log: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92488.log 2023-12-08T00:24:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92467.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-08T00:24:09.354 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92488.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.92574.log.gz 2023-12-08T00:24:09.354 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92969.log.gz 2023-12-08T00:24:09.363 INFO:teuthology.orchestra.run.smithi129.stderr: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91651.log 2023-12-08T00:24:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.92488.log.gz 2023-12-08T00:24:09.383 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117646.log 2023-12-08T00:24:09.383 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91651.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91651.log.gz 2023-12-08T00:24:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.117646.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.117646.log.gz 2023-12-08T00:24:09.465 INFO:teuthology.orchestra.run.smithi129.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92467.log.gz 2023-12-08T00:24:09.568 INFO:teuthology.orchestra.run.smithi184.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-08T00:24:09.854 INFO:teuthology.orchestra.run.smithi129.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.92769.log.gz 2023-12-08T00:24:10.510 INFO:teuthology.orchestra.run.smithi129.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92692.log.gz 2023-12-08T00:24:10.750 INFO:teuthology.orchestra.run.smithi184.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-08T00:24:10.963 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-08T00:24:12.879 INFO:teuthology.orchestra.run.smithi129.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-08T00:24:14.456 INFO:teuthology.orchestra.run.smithi184.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-08T00:24:14.636 INFO:teuthology.orchestra.run.smithi184.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-08T00:24:15.486 INFO:teuthology.orchestra.run.smithi129.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-08T00:24:15.497 INFO:teuthology.orchestra.run.smithi129.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-08T00:24:15.536 INFO:teuthology.orchestra.run.smithi184.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-08T00:24:15.565 INFO:teuthology.orchestra.run.smithi129.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-08T00:24:16.058 INFO:teuthology.orchestra.run.smithi184.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-08T00:24:16.060 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-12-08T00:24:16.060 INFO:teuthology.orchestra.run.smithi184.stderr:real 0m6.948s 2023-12-08T00:24:16.060 INFO:teuthology.orchestra.run.smithi184.stderr:user 0m25.366s 2023-12-08T00:24:16.060 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m0.880s 2023-12-08T00:24:16.114 INFO:teuthology.orchestra.run.smithi129.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-08T00:24:16.116 INFO:teuthology.orchestra.run.smithi129.stderr: 2023-12-08T00:24:16.116 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m7.038s 2023-12-08T00:24:16.116 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m32.795s 2023-12-08T00:24:16.116 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m1.233s 2023-12-08T00:24:16.117 INFO:tasks.ceph:Archiving logs... 2023-12-08T00:24:16.118 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/remote/smithi129/log 2023-12-08T00:24:16.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-08T00:24:20.514 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/remote/smithi184/log 2023-12-08T00:24:20.515 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-08T00:24:24.264 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-08T00:24:24.265 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-08T00:24:24.276 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c56ad3eaf0784383bf22c04e35d26e15c4baeafe/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-08T00:24:24.277 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-08T00:24:24.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-08T00:24:24.280 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-08T00:24:24.370 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-08T00:24:24.379 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-08T00:24:24.379 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T00:24:24.382 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T00:24:24.396 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2023-12-08T00:24:24.399 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2023-12-08T00:24:24.400 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-08T00:24:24.401 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2023-12-08T00:24:24.401 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +1769us[+1769us] +/- 69ms 2023-12-08T00:24:24.401 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 +58us[ +230us] +/- 53ms 2023-12-08T00:24:24.401 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -1648us[-1648us] +/- 64ms 2023-12-08T00:24:24.401 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T00:24:24.403 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-08T00:24:24.403 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2023-12-08T00:24:24.403 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +1752us[+1755us] +/- 69ms 2023-12-08T00:24:24.403 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 +129us[ +131us] +/- 53ms 2023-12-08T00:24:24.403 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -1665us[-1663us] +/- 64ms 2023-12-08T00:24:24.403 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T00:24:24.404 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-08T00:24:24.416 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-08T00:24:24.416 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-08T00:24:24.444 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-08T00:24:24.542 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-08T00:24:24.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-08T00:24:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T00:24:24.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-08T00:24:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T00:24:24.709 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-08T00:24:24.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-08T00:24:24.753 INFO:teuthology.task.internal:Duration was 3337.059779 seconds 2023-12-08T00:24:24.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-08T00:24:24.786 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-08T00:24:24.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-08T00:24:24.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-08T00:24:24.831 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-08T00:24:24.832 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-08T00:24:25.189 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-08T00:24:25.189 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2023-12-08T00:24:25.189 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-08T00:24:25.210 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2023-12-08T00:24:25.211 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-08T00:24:25.249 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-08T00:24:25.249 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-08T00:24:25.252 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-08T00:24:25.318 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-08T00:24:25.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-08T00:24:25.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-08T00:24:25.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-08T00:24:25.465 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-08T00:24:25.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-08T00:24:25.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-08T00:24:25.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-08T00:24:25.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-08T00:24:25.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-08T00:24:25.558 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2023-12-08T00:24:25.585 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2023-12-08T00:24:25.619 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-08T00:24:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T00:24:25.633 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-08T00:24:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T00:24:25.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-08T00:24:25.688 INFO:teuthology.task.internal:Transferring archived files... 2023-12-08T00:24:25.688 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/remote/smithi129 2023-12-08T00:24:25.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-08T00:24:25.743 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564/remote/smithi184 2023-12-08T00:24:25.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-08T00:24:25.785 INFO:teuthology.task.internal:Removing archive directory... 2023-12-08T00:24:25.786 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-08T00:24:25.788 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-08T00:24:25.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-08T00:24:25.846 INFO:teuthology.task.internal:Not uploading archives. 2023-12-08T00:24:25.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-08T00:24:25.870 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-08T00:24:25.870 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-08T00:24:25.873 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-08T00:24:25.885 INFO:teuthology.orchestra.run.smithi129.stdout: 265984 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 8 00:24 /home/ubuntu/cephtest 2023-12-08T00:24:25.887 INFO:teuthology.orchestra.run.smithi184.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 8 00:24 /home/ubuntu/cephtest 2023-12-08T00:24:25.888 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-08T00:24:25.903 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-08T00:24:26.045 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-08T00:24:26.045 INFO:teuthology.nuke:Checking targets against current locks 2023-12-08T00:24:26.070 DEBUG:teuthology.nuke:shortname: smithi129 2023-12-08T00:24:26.071 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-08T00:24:26.094 DEBUG:teuthology.nuke:shortname: smithi184 2023-12-08T00:24:26.095 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-08T00:24:26.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 23:20:29.167448', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk7r/sl5FVd4xb7HmAkal/vir+w88NJs5R3zxla6Lu2/yqjGtBg3JQ18Zye8EGgSE2uX+NVcIwyM+LRpgdP3FM='} 2023-12-08T00:24:26.121 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_21:35:03-rados-wip-yuri3-testing-2023-12-07-0727-distro-default-smithi/7482564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 23:20:29.165557', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA+p96ecYG4L+/ocaIhrFJzRHy5EQHa+ckbt4N23xqMZTSH7vVmpyFuMTVTnlnXutsIT1XFEzStsPoiZsgnm2k='} 2023-12-08T00:24:26.157 INFO:teuthology.orchestra.console.smithi129:Power off 2023-12-08T00:24:26.158 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-08T00:24:26.185 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2023-12-08T00:24:26.186 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T00:24:26.204 INFO:teuthology.orchestra.console.smithi184:Power off 2023-12-08T00:24:26.204 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-08T00:24:26.213 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-12-08T00:24:26.226 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2023-12-08T00:24:26.226 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T00:24:26.254 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2023-12-08T00:24:30.215 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T00:24:30.255 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T00:24:30.327 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-12-08T00:24:30.366 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2023-12-08T00:24:34.329 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T00:24:34.367 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T00:24:34.440 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2023-12-08T00:24:34.477 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2023-12-08T00:24:34.541 INFO:teuthology.orchestra.console.smithi129:Power off completed 2023-12-08T00:24:38.478 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T00:24:38.588 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2023-12-08T00:24:38.689 INFO:teuthology.orchestra.console.smithi184:Power off completed 2023-12-08T00:24:38.793 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3337.059778690338 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-12-08T00:24:38.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-08T00:24:38.871 INFO:teuthology.run:FAIL